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SUMMARY 


Operational  testing  is  an  integral  part  of  the  nateriel  acquisition 
cycle  in  the  Army  procurement  process.  It  is  oriented  towards  the 
evaluation  of  a developmental  item  under  realistic  conditions  as  part  of 
an  actual  troop  unit.  The  test  design  phase  is  an  essential  element  of 
operational  testing. 

In  order  to  facilitate  the  selection  of  measures  of  effectiveness 
used  in  these  tests,  the  critical  attributes  which  "best"  discriminate 
between  acceptable  and  unacceptable  systems  or  subsystems  need  to  be 
identified.  This  thesis  addresses  a method  which  provides  a basis  for 
the  se.lectlon  of  these  critical  attributes.  Once  these  attributes  are 
identified,  the  test  designers  of  subsequent  operational  tests  nay  use 
this  information  to  assist  them  in  the  test  design  phase. 

The  current  test  structure  in  operational  testing  is  not  ameanable 
to  the  standard  application  of  multivariate  statistics.  There  is  on.../  one 
replication  of  each  test  of  these  large  systems  and  the  data  collection 
procedure  precludes  direct  determination  of  relationships  among  the 
attributes.  Consequently,  the  methodology  developed  in  this  thesis 
encompasses  a means  to  combine  results  from  past  tests  with  subjective 
information  to  determine  the  relationship,  in  terms  of  covariances, 
between  each  two  attributes.  This  information  is  incorporated  with 
subjectively  obtained  acceptable  and  unacceptable  mean  vectors  in 
stepwise  discriminant  analysis. 
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It  is  concluded  th.at  Bultlvarlete  analysis  techniques  say  be 
a valuable  aid  in  determining  which  attributes  contribute  more  in 
distinguishing  between  successful  and  unsticcessful  systems.  It  .is  also 
concluded  that  the  current  test  design  for  operational  testing  can  be 
modified  to  facilitate  a broader  use  of  multivariate  statistical  analysis 
techniques.  This  modification  should  permit  (1)  the  correlations  among 
the  attributes  to  be  objectively  determined  and  (2)  the  marginal  normality 
of  observations  for  each  attribute  to  be  validated. 
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CHAPTER  I 
INTRODUCTION 

Background  of  the  Problem 

The  United  States  Army  Operational  Test  and  Evaluation  Agency 
(OTEA)  Is  a Department  of  the  Army  Field  Agency  under  the  Army  Chief  of 
Staff.  OTEA's  mission  Is  to  support  the  materiel  acquisition  and  force 
development  processes  by  (1)  exercising  responsibility  for  all  operational 
testing  (or),  (2)  managing  force  development  testing  and  experimentation 
(FDTE),  and  (3)  managing  joint  user  testing  for  the  army.  It  must  Insure 
that  user  testing  Is  effectively  planned,  conducted,  and  araluated  with 
emphasis  on  adequacy,  quality,  and  credlblllt> . It  actively  participates 
in  the  conduct  of  and  provides  Independent  evaluations  of  operationl  tests 
conducted  on  major  and  selected  nonmajor  systems,  as  well  as  major  FDTE 
and  other  systems  designated  by  appropriate  authority  (38). 

Operational  testing  Is  an  Integral  part  of  the  materiel  acquisition 
cycle.  It  is  oriented  towards  the  evaluation  of  a developmental  item 
under  realistic  conditions  as  part  of  an  actual  troop  unit.  The  purposes 
of  operational  testing  are:  (1)  evaluation  of  the  item's  desirability 

compared  to  equipment  already  in  the  inventory;  (2)  evaluation  of  mili- 
tary utility,  operational  effectiveness,  and  operational  suitability; 

(3)  assessment  of  the  need  for  modification;  and  (4)  assessment  of  the 
adequacy  of  organization,  doctrine,  and  tactics  (34). 
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Force  development  testing  and  experimentation  Involves  troop 
tests,  field  tests,  and  experiments  performed  by  or  for  the  users.  The 
tests  support  the  force  development  process  by  examining  the  Impact, 
potential,  or  effectiveness  of  selected  concepts,  doctrine,  organization, 
and  materiel  (34) . A test  can  support  the  materiel  acquisition  process 
by  providing  data  to  assist  in  the  establishment  of  the  required  opera- 
tonal  capability,  to  develop  fundamental  data  necessary  for  a full  under- 
standing of  the  performance  of  a materiel  system,  or  to  assist  in 
validating  doctrine  and  tactics  to  counter  threat  response  to  a system 
once  deployed  (38) . 

The  full  development  of  an  operational  test  from  the  initial 
planning  phase  through  the  final  test  report  is  a long  and  detailed  pro- 
cess. This  process  delineates  exactly  how  each  aspect  of  conducting  the 
tests  is  developed.  However,  the  aspect  of  prime  consideration  in  this 
paper  is  the  development  of  measurable  attributes,  i.e.,  measures  of 
effectiveness  (MOE),  of  the  tests. 

The  first  step  in  the  developmental  process  is  the  initial 
approach,  that  is,  the  listing  of  tentative  operatiorul  issues.  These 
issues  are  the  aspects  of  the  system's  capability  that  must  be  questioned 
before  the  system's  effectiveness  is  known.  They  are  broad  in  nature  and 
are  not  necessarily  directly  measurable.  These  tentative  operational 
issues  are  evaluated  and  critical  operational  issues  are  developed  on 
the  basis  of  relevance,  importance,  and  risk.  Finally,  the  critical 
issues  are  consolidated  as  necessary,  and  the  issues  for  operational 
testing  are  selected  after  considering  their  validity,  practicality,  and 


relative  costs. 


When  the  operational  Issues  have  been  refined,  statements  of  test 
objectives  are  developed.  These  statements  Identify  the  evidence  required 
to  address  particular  Issues.  The  test  objectives  may  still  not  be 
measurable:  hence,  they  must  be  divided  Into  subobjectives.  The  subob- 
jectives are  further  subdivided  Into  lower  levels  of  data  requirements 
until  measurable  requirements  emerge.  A data  requirement  is  finally  In 
a form  suitable  for  measurement  when  It  can  be  answered  by  a number. 

These  data  requirements,  MOE,  are  later  refined  In  the  Final  Test 
Design.  The  refinement  is  necessary  because  one  aspect  of  effectiveness 
often  has  several  possible  measures  and  not  all  are  needed.  The  questions 
of  redundancy  and  duplication  are  addressed,  the  advantages  and  disad- 
vantages are  considered,  and  a decision  is  made  as  to  which  measures  to 
employ  in  the  tests.  The  selection  of  measures  involves  some  risk  of 
selecting  inferior  measures:  consequently,  some  special  assistance  is 
needed  in  making  this  selection. 

This  special  assistance  can  come  in  the  form  of  specialists  who 
are  familiar  wiU-  doctrine,  organization,  human  factors,  logistics,  and 
threat,  and  in  the  form  of  information  developed  from  previous  operational 
tests. 

Normally,  operational  testing  considerations  begin  with  the 
development  of  the  test  item  and  conclude  with  the  publication 
of  the  final  report.  However,  it  should  be  noted  that  one  of 
the  paths  to  improved  operational  test  methodology  begins  after 
the  final  report  is  published.  That  is,  data  analyzed  in  the 
post-test-report  period  as  a means  >o  further  refine  the  nature 
of  influencing  factors  can  be  used  to  improve  the  state  of  t.ie 
art  for  subsequent  operational  testing.  Thus,  field  testing 
not  only  contributes  to  system  evaluation  objectives,  but  has 
th'.  potential  for  contributing  to  all  future  operational  tests. 

This  final  contribution  may  be  quite  as  important  in  the  long 
run  as  answering  the  test  objectives  (38). 
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Definition  of  the  Problem 

A key  area  of  current  Interest  to  OTEA  Is  the  evaluation  of 
tactical  command  and  control  systems.  One  accepted  definition  of  a 
tactical  command  and  control  system  is  an  arrangement  of  personnel,  fa- 
cilities, and  the  means  for  information  acquisition,  processing,  and 
dissemination  employed  by  a commander  in  planning,  directing,  and 
controlling  tactical  operations. 

The  introduction  of  sophisticated  computer-based  command  and 
control  systems  into  the  materiel  acquisition  process  raises  a problem 
in  the  operational  testing  of  such  systems.  In  the  past,  operational 
tests  have  been  able  to  evaluate  hardware  and  software  independently; 
however,  there  is  presently  a need  to  evaluate  the  operational  effective- 
ness of  the  entire  system.  This  system  consists  of  the  hardware,  soft- 
ware, and  personnel  interface  under  complex  operational  conditions.  The 
evaluation  should  take  into  account  the  interplay  of  all  relevant 
influencing  variables. 

The  general  problem  of  this  thesis  is  a current  requirement  for 
the  develop'.iient  of  a detailed  methodology  for  designing,  planning,  and 
evaluating  the  results  of  operational  tests  and  evaluations  of  complex 
command  and  control  systems.  The  specific  problem  of  this  thesis  impacts 
directly  on  the  design,  plan,  and  evaluation  of  operational  tests.  This 
problem  is:  How  can  the  size  of  these  complex  tests  be  reduced  without 

reducing  the  amount  ot  information  about  the  system  being  tested? 

Purpose  of  this  Thesic 


The  purpose  of  this  thesis  is  to  develop  a methodology  which  will 
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provide  a rational  basis  for  selecting  critical  attributes  of  complex 
commard  and  control  systems.  This  selection  process  will  permit  the 
deletion  of  the  evaluation  of  non-crltical  attributes  from  subsequent 
operational  tests. 


Review  of  Literature 

The  practical  use  of  multivariate  statistics  to  determine  the  crit- 
icality of  attributes  in  operational  tests  is  virtually  untrodden  ground. 
Both  computer  based  and  manual  literature  searches  revealed  no  direct 
references  to  this  area  of  study.  There  are,  however,  many  articles  and 
books  about  multivariate  statistics.  The  multivariate  techniques  of  prin- 
cipal components  analysis  (3,  4,  9,  15,  30),  factor  analysis  (3,  4,  6, 

9,  15,  30),  discriminant  analysis  (3,  4,  9,  17,  18,  19,  21,  30),  and 
cluster  analysis  (1,  5,  7,  16,  20,  23,  2'^,  5,  32,  36,  40)  were  considered 
as  possible  techniques  to  be  utilized.  All  of  these  techniques  were 
eliminated  from  consideration  because  of  the  design  of  the  tests.  The 
methodology,  as  will  be  seen  later,  does  not  provide  for  any  replications 
of  th«.  test.  The  operational  tests  are  so  complex  that  the  extremely 
high  costs  preclude  replications.  Multiple  classification  analysis,  the 
Automatic  Interaction  Detection  System,  was  considered,  but  was  eliminated 
because  of  the  requirement  for  a large  sample  of  data  (38) . 

Interviews  and  extended  discussions  with  representatives  of  the 
Methodology  Branch,  Test  Design  Division,  Operational  Test  and  Evaluation 
Agency  and  the  Test  and  Evaluation  Division,  Command  Control  and  Communi- 
cations Directorate,  Modem  Army  Selected  Systems  Evaluation  and  Review 
resulted  in  valuable  Insight  into  the  problem  area.  However,  techniques 
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that  could  be  modified  or  utilized  directly  were  not  available  from  these 
sources.  The  relative  lack  of  prior  study  In  this  field  and  the  nature 
of  the  tests  themselves  led  to  the  development  of  the  methodology  presented 
In  Chapter  III. 


General  Approach  and  Overview 

The  multidimensional  aspect  of  complex  systems  lends  Itself  to 
the  application  of  multivariate  analysis.  The  complexity  of  these  systems 
Inherently  causes  the  tests  that  are  used  to  evaluate  the  systems  to  be- 
come large  and  unwieldy.  Chapter  II  will  address  the  methodology  of  these 
tests.  It  will  emphasize  the  design  and  evaluation  procedure  for  the 
tests. 

Chapter  I 'I  will  re\’lew  the  analysis  leading  to  the  selection  of 
the  methodology.  The  detailed  procedure  for  each  step  of  cne  methodology 
to  establish  the  criticality  of  attributes  will  then  be  presented  In 
Chapter  IV. 

An  Integral  part  of  this  methodology  is  the  use  of  a computer  to 
facilitate  the  manipulation  of  the  multivariate  data.  The  computer  pro- 
grams used  will  be  discussed  in  Chapter  IV  in  the  sequence  in  which  they 
are  utilized. 

A demonstration  of  the  methodology  will  be  presented  in  Chapter  V 
in  order  to  illustrate  the  entire  procedure.  The  final  conclusions  and 
recommendations  will  be  presented  in  Chapter  VI o 
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CHAPTER  II 

EXISTING  TEST  STRUCTURE 
Methodology  of  the  Tests 

To  determine  critical  attributes  of  operational  tests,  it  is 
necessary  to  understand  the  intricacies  of  those  tests.  Each  test  is 
different  from  every  other  test  in  that  the  attributes  which  are  measured 
vary  according  to  the  operational  issues  involved.  Although  specific 
tests  differ,  the  methodology  by  which  those  tests  are  evolved  from 
operational  issues  to  conclusion  is  similar  for  a large  set  of  tests. 

Normally,  OTEA  provides  a Final  Test  Design  Plan  for  each  test 
prior  to  the  start  of  detailed  planning  at  the  test  site.  Exceptions 
occur  for  tests  that  are  executed  for  OTEA  by  Modem  Army  Selected  Systems 
Evaluation  and  Review  (MASSTER) . In  this  latter  case,  the  Final  Test 
Design  Flan  is  prepared  by  MASSTER  for  OTEA  approval.  The  test  structure 
examined  in  this  chapter  Illustrates  the  general  methodorogy.  A specific 
test  is  used  as  an  example  in  order  to  concisely  Illustrate  the 
methodology. 

The  Division  Command  Post  Test  (Test  Number  FM  286)  was  selected 
as  the  example.  Its  purpose  is  to  evaluate  a proposed  division  command 
post  (CP)  system  and  thus  is  a test  of  a command  and  control  system.  The 
methodology  presented  in  FM  286  is  representative  of  the  general  method- 
ology used  by  GTE.'.  The  data  from  this  test  are  not  classified  and  thus 
are  available  for  analysis.  Because  the  test  was  conducted  in  January 
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1975,  some  of  the  principles  Involved  In  the  test  are  available  to 
explain  unanswered  questions  that  are  not  covered  In  the  written  plans 
aiui  reports.  The  test  structure  presented  In  this  chapter  Is  derived 
from  (1)  the  Detail  Plan  for  Execution  (FM  286)  (10) , (2)  the  Division 
Conmand  Post  Test  Report  (FM  286)  (11),  (3)  the  MASSTER  Test  Officer's 
Planning  Manual  (34),  and  (4)  Interviews  with  representatives  of  MASSTER. 
In  the  Interest  of  clarity,  the  administrative  details  of  the  test  will 
not  be  related,  but  they  are  available  In  the  references  mentioned  above 
for  the  Interested  reader. 

Pattern  of  Analysis 

The  purpose  of  the  test  FM  286  Is  to  evaluate  a propose>i  division 
command  post  (CP)  system.  The  results  of  the  test  are  to  be  utilized 
to  support  reconmedatlons  concerning  tactical  organization,  equipment, 
and  command  and  control  doctrine  and  procedures.  They  will  be  the  basis 
for  subsequent  changes  to  Tables  of  Organization  and  Equipment. 

The  objectives  that  were  derived  from  the  operational  Issues  men- 
tioned In  Chapter  I are  the  evaluation  of  the  efficiency  of  the  command 
post  In  command  and  control  of  division  tactical  operations  and  the 
evaluation  of  the  vulnerability  of  the  command  post  during  division  tac- 
tical operations.  Efficiency  is  defined  as  "a  measure  of  the  degree  to 
which  a system  performs  a set  of  defined  tasks  or  mission  requirements." 
(10)  Vulnerability  Is  defined  as  "a  measure  of  the  susceptibility  of 
the  command  and  control  system  to  any  reasonable  means  through  which  its 
combat  effectiveness  might  be  reduced."  (10)  Note  that  the  system  is 
being  evaluated  and  not  the  performance  of  the  players  within  the  system. 
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For  the  purpose  of  coatinulty,  the  development  of  objectxve  1,  efficiency, 
is  pursued  and  the  development  of  objective  2,  vulnerability,  is  omitted. 

Amy  Regulation  310-25  defines  a system  as  "an  integrated  relation- 
ship of  compouents  aligned  to  establish  functional  continuity  toward  the 
successful  performance  of.  a defined  task  or  tasks."  Therefore,  the 
division  command  and  control  system  was  divided  into  the  subsystems  of 
command,  operations,  intelligence,  and  combat  service  support. 

The  primary  functions  of  the  command  subsystem  are  the  management 
functions  of  planning,  organizing,  directing,  coordinating,  and  con- 
trolling. In  order  to  find  a measure  of  efficiency  for  the  command  sub- 
system, a measure  of  efficiency  for  each  of  the  functions  listed  above 
musL  be  found.  Thus  the  management  functions  of  the  command  subsystem 
become  data  requirements.  The  data  requirements  are  not  measurable. 
Consequently,  they  must  be  divided  and  subdivided  until  measurable  data 
requirements  are  developed.  An  abbreviated  pattern  developed  in  this 
manner  is  shown  in  Figure  1. 

The  primary  staff  functions  are  preparing  plans,  orders,  and  re- 
ports; providing  Information;  and  supervising  the  execution  of  plans  and 
orders.  Considering  operations.  Intelligence,  and  combat  service  support 
as  primary  staff  functions,  each  subsystem  is  subdivided  according  to 
these  functions.  Since  all  subsystems  have  the  same  function,  only  the 
operations  subsystem  will  be  pursued.  As  in  the  command  subsystem,  the 
functions  of  the  operations  subsystem  are  unmeasurable  data  requirements 
that  must  be  further  subdivided  until  measurable  data  requirements  are 
developed.  An  abbreviated  pattern  developed  in  this  manner  is  shown  in 
Figure  2.  The  Intelligence  and  combat  service  support  subsystems  are 
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Figure  1.  Abbreviated  Pattern  of  Analysis  - Objective  1. 
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Figure  2.  Abbreviated  Fattem  of  Analysis,  Operations. 


analogous  to  that  for  operations. 

The  neasurea  o^  effectiveness  CMOE)  that  were  used  to  develop  the 
pattern  of  analysis  are  a set  of  dependent  or  resfonse  variables  which 
demonstrate  the  adequacy  of  the  command  post  system  to  accoiq>llsh  mission 
requirements  under  specific  conditions.  These  specific  conditions  are 
the  independent  variables. 

Evaluation  Plan 

The  evaluation  effort  for  objective  1 was  essentially  subjective. 
It  was  .reinforced  with  quantitative  data  where  feasible.  This  subjective 
evaluation  was  based  primarily  on  the  end-of-test  reports  prepared  by 
the  evaluators  and  players.  Extensive  use  of  rating  type  questions, 
using  semantic  differentially  scaled  responses,  were  used  in  the  dally 
questionnaire.  These  ratings  allowed  the  opinions  of  the  players  and 
evaluators  to  be  quantified  on  a dally  basis  and  subsequently  aggregated 
within  each  major  staff  section.  When  aggregated,  the  daily  ratings  also 
provided  the  test  analyst  with  numerical  performance  indicators  at  each 
level  of  the  analysis.  Objective  measurements,  such  as  accuracy  of  maps 
and  charts,  are  also  included  in  the  assessment  of  dependent  variables. 
All  of  these  types  of  input  provide  a means  to  assess  the  adequacy  of 
the  command  post  concept  to  accomplish  mission  requirements. 

A subjective  extension  of  the  player  and  evaluator  semantic  dif- 
ferential responses  Is  an  ad equate/ Inadequate  evaluation  scheme.  This 
Is  the  basis  for  the  development  of  criteria  used  by  the  test  analyst  to 
assess  the  measure  of  performance  (MOP)  for  the  command  post  concept. 
Criteria  for  the  development  of  conclusions  for  objective  1 are  defined 
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in  the  following  naimer: 

Adequate^  Combined  ratings  and  rationale  demonstrated  that  the 
section  could  accomplish  functions  or  mission  requirements 
under  test  conditions. 

Borderline-  Although  ratings  and  rationale  may  have  demonstrated 
that  the  section  rould  accomplish  the  functions  or  requirements, 
there  were  negative  ratings  and  rationale  that  detracted  from  the 
overall  ability  to  accomplish  mission  requirements.  These  nega- 
tive factors  had  to  be  correctable  without  major  organizational 
or  functional  changes  to  the  OFM  or  SOP. 

Inadequate-  Ratings  and  rationale  demonstrated  that  the  section 
could  not  accomplish  mission  requirements.  Major  organizational 
or  functional  changes  had  to  be  made  to  either  the  OFM  or  SOP 
(11). 

OFM  and  SOP  are  acronyms  for  organizations  and  functions  manual  and 
standing  operating  procedures,  respectively. 

To  enhance  the  evaluation  plan,  objective  1 is  functionally  di- 
vided into  areas  for  which  findings  are  generated.  Figure  3.  Note  that 
this  functional  division  breaks  down  each  subsystem  into  a section  that 
is  operationally  responsible  for  the  subsystem.  The  elements  within 
that  section  are  designated.  These  elements  comprise  the  level  at  which 
the  measures  of  performance  are  evaluated. 

The  data  collection  plan  involves  both  players  and  evaluators 
in  obtaining  data  on  MOE's  presented  in  the  plan  of  analysis.  The  eval- 
uators are  screened  to  Insure  their  qualifications  and  to  Insure  the 
credibility  of  their  observations  and  evaluations  based  on  grade,  mil- 
itary occupational  speciality  (MOS),  command  and  staff  experience,  mil- 
itary schooling,  etc.  There  are  three  categories  of  evaluators.  Cate- 
gory 1 consists  of  officers  in  the  grades  of  05  and  06.  They  are 
assigned  to  the  subsystem-section  level  to  evaluate  the  effect  of  the 
organizational  concept  on  staff  and  section  performance  by  observation 
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Figure  3.  Functional  Areas  Used  in  Evaluation,  Objective  1 
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and  review  of  staff  outputs.  Category  2 evaluators  consist  of  officers 
in  the  grades  of  03  and  04.  They  are  assigned  to  the  elemer.c  level  to 
evaluate  the  performance  of  one  or  more  selected  staff  elements  and  to 
record  staff  performance  as  appropriate.  Category  3 consists  of  enlisted 
men  in  the  grades  of  E6  and  E7.  They  are  assigned  to  the  element  level 
to  collect  data  from  selected  maps  and  charts,  prepare  CP  layout  sketches, 
inventory  major  items  of  equipment,  and  record  displacement  data. 

There  are  seventeen  types  of  questionnaires  and  data  forms  that 
are  usea  by  the  players  and  evaluators.  The  questionnaires  and  data 
forms  are  tailored  as  to  the  data  source,  frequency  of  submission,  and 
level  of  required  detail.  These  are  completed  and  submitted  according 
to  the  Data  Collection  Plan,  Appendix  E to  the  DPE. 

Corresponding  to  the  categories  of  evaluators  are  players  that 
complete,  respectively,  the  same  questionnaires  and  data  forms.  Thus, 
there  is  a dual  rating  system.  The  data  from  the  players  and  evaluatjrs 
is  assimilated  according  to  the  Data  Reduction  Plan,  Appendix  F of  the 
DPE  for  FM  286.  There  are  many  details  in  the  reduction  plan;  however, 
the  salient  feature  is  the  method  by  which  player  and  evaluator  observa- 
tions arc  combined  in  order  to  formulate  an  evaluation.  Figure  4 presents 
an  overview  of  the  interaction  of  player-evaluator  response. 

Prior  to  a discussion  on  data  reduction,  it  is  necessary  to  explain 
the  acrony!"  EEA.  EEA,  essential  elements  of  analysis,  are  those  data  re- 
quirements that  have  been  developed  for  a specific  test.  The  first  level 
PEA  corresponds  to  the  subsystems;  the  second  level  EEA  corresponds  to 
the  functions;  and  the  remaining  levels  of  EEA  correspond  respectively 
to  the  succeeding  subdivisions. 


Figure  4.  Evaluation  Methodology  Concept  for  Object! 
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The  ratings  for  the  third  level  EEA  (e.g.,  1.2. 2. 2,  Adequacy  of 
Information)  are  basically  obtained  in  the  following  steps: 

Step  1:  Ratings  of  fourth  le.'el  EEA  are  made  by  each  category  2 and  3 

evaluator  and  player  for  each  element.  These  ratings  are  based 
on  a five-point  adjectival  rating  scale  where  1 is  considered 
best  and  5 is  considered  worst  (e.g.,  1 corresponds  to  adequate, 
3 corresponds  to  borderline,  and  5 corresponds  to  Inadequate). 
The  frequency  of  observations  that  fall  in  the  five  rating 
categories  for  each  element  and  data  requirement  are  recorded 
separately  for  both  evaluators  and  players.  If  an  observation 
is  placed  in  rating  category  1,  it  receives  a weighted  value  of 
1.  If  an  observation  is  placed  in  rating  category  2,  it 
receives  a weighted  value  of  2.  This  procedure  extends  analo- 
gously for  the  remain! ..g  three  rating  categories.  If  an  obser- 
vation is  not  made,  a sixth  rating  is  available.  This  rating 
is  considered  as  having  no  weight  and  does  not  affect  the 
evaluation  procedure  (See  Table  1) . 

Step  2:  The  fourth  level  ratings  are  summed  across  the  rating  categories 
for  each  element  of  the  data  requirement.  The  frequencies  of 
the  rating  scores  are  multiplied  by  their  respective  weights 
and  an  element  score  is  obtained  by  dividing  the  sum  of  these  by 
tie  total  number  of  observations  by  that  element.  This  is  done 
for  both  players  and  evaluators  at  this  level  (See  Table  2) . 

Step  3:  A third  level  EEA  score  is  obtained  for  each  element  by  taking 

a grand  average  across  the  fourth  level  EEA  (data  requirements). 
This  is  done  for  players  and  evaluators  (See  Table  3). 


Example  of  Element  Score  for  OPS: 


6(1)  + 4(2)  + 0(3)  + 0(4)  + 0(5)  „ 1.4 


10 


20 


Table  3.  Third  Level  Element  Score 
(Category  2 and  3) 

Data  Requirement  1.2. 2. 2. 


PLAYER  ELEMENT  SCORES /NUMBER  OF  0BSERVATI(»IS 

ELEMENT 

1.2.2. 2.1 

1.2.2. 2.2 

1.2. 2. 2.3 

1.2. 2. 2. 

OPS 

1.76/37 

1.84/37 

i. 86/86 

1.99/110 

FS 

2.10/10 

2.20/10 

3.00/10 

2.43/30 

TACP 

1.20/50 

2.00/50 

4.40/5 

2.53/15 

PLANS 

I.6O/15 

1.87/15 

4.07/15 

2.51/45 

FSE 

1.30/20 

1.48/25 

3.40/20 

2.0  V 65 

DAiffi 

1.72/i5 

2.20/20 

4.36/25 

2.66/76 

LNO 

1.00/50 

1.00/50 

2.60/50 

1.53/15 

Example  of  Third  Level  Element  Score  for  OPS: 

1.78(37)  + 1.84(37)  + 2.36(36)  , ^ 95 

no 


EVALUATOR  ELEMENT  SCORES /NUMBER  OF  OBSERVATIONS 

ELEMENT 

1.2.2. 2.1 

1.2. 2. 2. 2 

1.2. 2. 2. 3 

1.2. 2. 2. 

OPS 

1.40/10 

1.40/10 

1.80/ 10 

1.53/30 

FS 

1.30/10 

1.30/10 

2.10/10 

1.57/30 

TACP 

I.8O/1O 

2.00/10 

3.40/10 

2.40/30 

PLANS 

1.70/10 

2.00/10 

3.90/10 

2.53/30 

FSE 

1.40/10 

1.60/10 

3.10/10 

2.03/30 

DAME 

1.90/10 

2.30/10 

4.60/10 

2.93/30 

LNO 

- 

- 

- 

Example  of  Third  Level  Element  Score  for  OPS; 
1.4(10)  + 1.4(10)  + 1.8(10)  ^ 53 


30 
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Step  4:  Category  1 players  and  evaluators  make  independent  observations 
on  the  third  level  EEA.  A simple  average  Is  obtained  for  each 
(See  Table  4). 

Step  5:  A grand  average  Is  taken  across  the  elements  of  third  level 

EEA  scores  to  obtain  a third  level  EEA  score.  This  Is  done  for 
both  players  and  evaluators.  D.  R.  1.2. 2. 2. 

Player 

1.99(110)+2.43(30H2.53(15H2.51(45)+2.01(65H2. 80(70)1-1.53(15)  - 2.26 

350 

Evaluator 

1.53(30)1-1. 57(30)->-2.4(30)->-2. 53(30)1-2. 03(30) ^2. 93(30)  • 1.91 

180 

Step  6:  A mean  is  then  obtained  of  the  score  for  the  category  2 and  3 

players  (Step  5j  and  the  score  for  the  category  1 player  (Step  4). 


D.  R.  1.2. 2. 2. 

2.261-3.4  - 2.83 
2 

A score  for  the  evaluators  is  obtained  In  a similar  manner. 

1.91+2.0  - 1.96 


2 
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Table  4.  Third  Level  Elesaent  Score 
(Category  1) 

Data  Requirement  1.2. 2. 2. 


PLAYER  RATINGS 


ELEMENT 


G3 


Category  1 Third  Level  Score: 

0(1)  + 2(2)  + 1(3)  + 0(4)  + 2(5)  . 3., 


EVALUATOR  RATINGS 


ELEMENT 


G3 


Category  1 Third  Level  Score: 

1(1)  + 1(2)  + 2(3)  + 1(4)  + 0(5)  . 2. 
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Step  7:  The  mean  of  the  player  and  evaluator  scores  is  the  overall 

rating  for  the  third  level  EEA. 

2.83+1.96  » 2.39 
2 

Once  all  of  the  overall  scores  for  the  essential  elements  of 
analysis  are  obtained,  an  Independent  evaluator  considers  all  Input,  8ub> 
j active  and  objective,  and  formulates  a final  rating  for  each  third  level 
EEA.  llie  minimum  acceptable  standard  at  this  level  is  "borderline." 
Ratings  of  acceptable,  borderline,  or  unacceptable  are  established  for 
each  section.  Each  of  the  staff  sections  Is  then  subjectively  evaluated 
at  progressively  higher  levels  of  EEA  as  adequate  or  Inadequate  In  the 
performance  of  Its  primary  functions.  A rating  of  adequate  Is  the 
minimum  acceptable  standard  at  higher  levels  of  evaluation. 

Analysis  of  the  Test  Structure 

The  purpose  of  this  analysis  is  not  to  evaluate  the  methodology 
underlying  the  structure  of  the  test.  Rather,  the  purpose  is  to 
examine  the  salient  features  of  the  test  In  order  to  better  understand 
the  relationship  of  the  data  requirements  to  the  test  structure. 

The  nature  of  the  test  Is  highly  subjective.  The  objective 
inputs  do  assist  In  the  final  evaluation;  however,  the  subjective  inputs 
obtained  by  the  use  of  the  semantic  differentially  scaled  responses  and 
general  observations  have  a greater  impact  upon  the  final  evaluation  of 
the  system.  The  objective  inputs  and  subjective  inputs  sing  the  five- 
point  adjectival  scale  do  provide  a profile  of  the  individual  measurable 
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data  requlr£Dect8.  Tney  provide  a frequency  distribution  of  the  nuid>er 
of  times  the  data  requirements  were  observed  In  a particular  status, 
i.e. , the  nuid)er  of  observations  that  fell  in  the  rating  categories  1 
through  5.  The  manner  in  'hlch  the  observations  were  taken  precludes 
the  determination  of  relationships  between  data  requirements  (correla- 
tion). The  number  of  total  observations  and  time  of  observations  var- 
ied greatly.  The  range  of  total  observations  per  data  requirement  is 
Illustrated  by  the  data  used  In  the  demonstration  of  Chapter  4. 

The  Intervals  of  the  rating  scale  restrict  the  range  of  obser- 
vations. If  an  observer  evalxiates  a particular  measurable  data  require- 
ment as  slightly  above  borderline,  but  not  totally  adequate,  he  has  only 
one  alternative.  That  al tentative  is  to  assign  a rating  of  2.  The 
rating  of  2 reflects  that  the  data  requirement  is  evaluated  exactly  half- 
way between  borderline  and  adequate.  This  indicates  that  the  range  of 
the  rating  scale  might  need  to  be  extended.  The  optimal  number  of 
intervals  between  which  an  observer  can  discriminate  in  making  an 
evaluation  is  an  open  question  for  study. 

The  number  of  measurable  data  requirements  in  this  type  of  test 
Is  quite  large.  Since  multiple  observations  are  being  made  for  each 
measurable  data  requirement,  the  size  of  the  problem,  computationally, 
can  easily  get  out  of  control.  The  level  at  which  the  data  requlrem:^ats 
are  measurable  also  varies.  Thus  there  Is  the  added  problem  of  comparing 
data  requirements  from  different  level  EEA. 

The  evaluation  procedure  is  based  upon  the  elements  at  each  level 
rather  than  the  measurable  data  requirements.  As  seen  earlier,  evalu- 
ations at  each  level  EEA  are  made  that  reflect  the  status  of  the  elements. 
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The  observations  of  the  data  requirements  are  not  carried  forward.  This 
is  compounded  by  the  means  by  which  the  overall  evaluations  are  computed. 
The  averaging  process  dilutes  the  rating  category  S observations.  By 
Che  time  the  final  third  level  EEA  are  calculated,  the  "worse**  evalu- 
ations are  diluted.  Th^a  situation  indicates  a need  for  a method  to 
consider  all  observations  for  each  data  requirement. 
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CHAPTER  III 

REVIEW  OF  POTENTIAL  MULTIVARIATE  ANALYSIS  TECHNIQUES 

Introduction 

The  purpose  of  this  thesis,  as  stated  in  Chapter  I,  is  to  develop 
a methodology  which  will  provide  a rational  basis  for  the  selection  of 
critical  attributes  of  complex  command  and  control  systems.  Chapter  II 
presented  an  actual  test  that  Illustrates  the  methodology  that  is  cur- 
rently used  to  evaluate  complex  command  and  control  systems.  In  this 
chapter  the  term  "critical  attributes"  will  be  defined  and  potential 
multivariate  analysis  techniques  for  determining  critical  attributes 
will  be  evaluated. 

The  attributes  of  a system  may  be  defined  in  several  ways.  They 
may  be  an  Integral  part  of  the  evaluation  procedure  used  in  a particular 
test.  In  the  test  discussed  above,  the  evaluation  procedure  Included 
observing  the  measurable  data  requirements,  formulating  a value  for  each 
element,  and  carrying  the  element  evaluation  up  the  levels  of  EEA  in 
order  to  obtain  section  evaluations.  Here  the  elements  could  be  con- 
sldo.red  to  be  the  attributes.  In  that  procedure,  the  observations  made 
on  the  specific  data  requirements  were  lost  in  the  evaluation  process. 

The  attributes  may  also  be  defined  as  the  measurable  data  require 
ments  of  the  tests.  The  evaluation  procedure  may  vary  between  two  tests 
however,  the  methodology  by  which  operational  Issues  are  subdivided 
into  measurable  data  requirements  is  stable.  If  the  measurable  data 
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requirements  are  considered  as  the  attributes,  all  observations  have  a 
visible  impact  on  the  overall  resulting  evaluation  up  to  the  third  level 
EEA..  In  order  to  insure  the  greatest  degree  of  applicability  for  the 
methodology  presented  here,  the  term  "attribute"  will  be  construed  to 
mean  measurable  data  requirements. 

The  critical  attributes,  or  critical  measurable  data  requirements, 
of  a system  are  those  attributes  which  impart  maximum  information  to  its 
evaluation.  In  the  final  analysis,  the  evaluation  of  a system  classifies 
the  system  as  either  acceptable  or  unacceptable.  Jhus,  the  attributes 
which  contribute  most  to  deciding  whether  a system  is  acceptable  or  un- 
acceptable are  classified  as  critical  and  those  which  contribute  least 
are  classified  as  noncritlcal.  The  degree  of  criticality  or  noncriti- 
cality is  dependent  upon  a given  situation,  or  in  this  case,  upon  a given 
operational  test.  As  will  be  seen  later  in  this  chapter,  the  degree  of 
criticality  can  be  controlled  depending  on  the  parameters  used  in  the 
methodology  presented  herein. 

In  developing  a methodology  to  determine  critical  attributes,  the 
practical  use  of  the  methodology  was  the  foremost  consideration.  The 
extreme  costs  of  opexatlon/il  tests  preclude  replications  and  hence  pre- 
clude the  standard  multivariate  analysis  techniques  such  as  principal 
component  analysis,  factor  analysis,  discriminant  analysis,  and  cluster 
analysis.  The  procedure  developed  had  to  take  into  consideration  the 
test  design  process,  the  structure  of  the  test,  and  the  form  of  the  data. 

The  central  theme  of  the  design  process  was  the  development  of 
measurable  data  requirements  from  the  initial  operational  Issues.  In 
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tbe  design  phase,  once  all  measurable  data  requirements  (attributes) 
were  formulated,  a subjective  process  delineated  those  attributes  con- 
sidered necessary.  This  process  took  into  conslderaticn  the  possible 
necessity  of  obtaining  redundant  informacion  by  taking  manual  measures 
to  beck  up  sophisticated  instrumentation.  It  also  took  into  account  the 
need  for  duplication  of  measures.  This  Includes  observing  two  attributes 
that  measure  the  same  thing.  The  great  degree  of  work  and  expertise  that 
went  into  making  the  decisions  of  the  Initial  attributes  should  be  used 
to  the  maximum  extent  possible. 

The  structure  of  the  test  should  be  considered.  The  dendritic 
breaks  down  the  data  requirements  into  distinct,  yet  related,  levels  of 
essential  elements.  In  FM  286,  (see  Figure  2),  the  operations  subsystem 
was  broken  down  into  three  major  areas  of  consideration:  preparing  plans, 

orders,  and  reports;  providing  Information;  and  supervising  plans  and 
orders.  These  in  turn  were  subdivided.  The  possible  advantageous  use  of 
the  structure  should  be  considered  in  the  development  of  a methodology. 

The  form  of  the  data  should  also  be  considered.  Initially,  the 
data  is  compiled  in  the  form  of  frequency  distributions.  If  the  data  is 
used  at  the  attribute  level,  a more  accurate  picture  can  be  obtained  of 
the  attributes  under  consideration. 

The  methodology  presented  here  evolved  from  a detaixed  study  of 
FM  286,  the  considerations  given  above,  and  review  of  multivariate  anal- 
ysis theory  and  associated  techniques.  The  general  methodology  will  be 
presented  in  the  following  section.  A more  detailed  discussion  will 
then  be  presented  that  explains  the  procedures  and  techniques  in  greater 


detail. 
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Applicability  of  Multivariate  Techniques 

Considering  the  definition  of  critical  attributes  and  the  deline- 
ation of  the  criteria  for  selecting  a methodology  that  were  derived  from 
the  existing  test  structure,  the  task  of  finding  a methodology  for  selec- 
ting critical  attributes  of  a system  is  somewhat  facilitated.  The  form 
of  the  data  on  the  attributes  is  a frequency  distribution  with  observa- 
tions on  a range  of  discrete  numbers  from  1 to  5.  There  is  only  one 
replication  of  the  test  and  the  time  interval  for  reporting  observations 
is  so  large  that  statistical  correlations  cannot  be  obtained.  Thus, 
there  are  three  alternatives  open  for  investigation. 

A totally  subjective  methodology  can  be  developed  that  involves 
no  statistical  inference.  This  alternative  is  considered  infeasible  be- 
cause of  the  nature  of  the  problem.  The  number  of  variables  is  so  great 
that  the  time  involved  would  extend  the  design  phase  past  reasonable 
suspense  dates.  The  number  of  people  necessary  for  a totally  subjective 
method  could  easily  exceed  the  number  available.  Here,  it  must  be  under- 
stood that  the  design  phase  of  one  test  may  be,  and  probably  is,  con- 
ducti.'i  simultaneously  with  a number  of  other  tests.  An  additional  factor 
is  the  fact  chat  the  degree  of  criticality  is  difficult  to  define  subjec- 
tively. For  these  reasons,  the  totally  subjective  approach  was  eliminated 
from  consideration. 

A second  approach  might  be  totally  objective,  represented  by 
traditional  multivariate  statistical  techniques.  Since  there  is  only 
one  replication  of  the  test,  initiating  a purely  statistical  approach 
would  require  the  simulation  of  a multiple  set  of  data.  The  genera- 
tion of  the  data  would  have  to  be  accomplished  by  generating  sets  of 
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independent  data  for  each  attribute.  This  would  have  to  be  done  because 
a covariance  matrix  and  mean  vector  are  necessary  to  generate  a multi- 
variate distribution.  The  sets  of  Independently  generated  data  for  the 
attributes  would  not  necessarily  produce  reliable  statistical  Inference. 

The  third  approach  is  the  use  of  subjective  input  in  conjunction 
with  multivariate  analysis  techniques.  The  objection  to  the  totally  ob- 
jective approach  was  the  lack  of  a covariance  matrix  and  mean  vector  from 
which  the  generation  of  multivariate  observations  could  be  facilitated. 

If  a covar.  lance  matrix  and  mean  vector  can  be  developed  using  the  avail- 
able data  and  subjective  information,  then  the  generation  of  multivariate 
observations  is  feasible. 

At  this  point  in  the  development  of  the  methodology,  it  is 
assumed  that  a mean  vector  and  covariance  matrix  can  be  subjectively 
produced.  Having  made  this  assumption,  the  available  multivariate  analy- 
sis techniques  will  be  reviewed  in  order  to  find  a technique  that  can 
be  utilized  to  distinguish  critical  attributes. 

In  selecting  an  appropriate  technique  applicable  in  the  context 
of  operational  tests,  there  are  two  constraints  that  restrict  the  flexi- 
bility of  technique  selection.  The  first  constraint  is  that  a technique 
is  sought  whereby  the  attributes  that  contribute  least  to  the  evaluation 
of  the  system  are  delineated.  This  is  equivalent  to  specifying  those 
attributes  that  contribute  the  most  to  the  evaluation.  A second  con- 
straint is  that  the  critical  attributes  are  those  attributes  which  best 
establish  whether  a system  is  acceptable  or  unacceptable.  Thus,  the  tech- 
nique should  involve  distinguishing  between  acceptable  and  unacceptable 
populations. 
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The  first  constraint  reduces  the  field  of  multivariate  analysis 
techniques  to  four  areas:  principal  conq>onent  analysis » factor  analysis, 

cluster  analysis,  and  discriminant  analysis.  Principal  coiq>onent  analy- 
sis, largely  attributed  to  Hotelling  (4),  deals  with  the  coordinate  struc- 
ture of  multivariate  observations.  This  technique  seeks  to  make  linear 
combinations  of  the  variables  (principal  components)  such  that  each  of 
the  linear  combinations  captures  as  much  variation  in  the  vector  of  vari- 
ables as  possible.  At  the  same  time,  each  principal  component  is  formu- 
lated so  that  it  is  linearly  Independent  of  all  the  other  principal 
components.  Those  principal  components  that  contribute  most  to  the  vari- 
ance would  then  be  the  critical  variables.  This  technique  is  rejected 
for  basically  two  reasons.  The  first  reason  is  that  the  definition  of 
the  term  "critical  variable"  does  not  coincide  with  the  given  definition 
of  critical  attribute.  The  second  is  that  principal  component  analysis 
deals  with  one  of  variables  and  one  population.  The  desired  technique 
aus^  involve  two  populations. 

Factor  analysis  is  an  extension  of  principal  component  analysis. 

It  is  a procedure  for  reducing  complexity  of  correlational  data.  From 
the  original  set  of  variables  under  consideration,  it  selects  a smaller 
set  of  orthogonal  reference  axes  to  span  the  original  data.  This  tech- 
nique Is  also  eliminated  from  consideration.  Factor  analysis,  like 
principal  components,  addressess  one  set  of  variables  and  one  population. 

Cluster  analysis  is  a process  of  sorting  entities  into  categories 
according  to  their  overall  slmularitles  by  comparing  vectors  of  variables. 
In  cluster  analysis,  very  little  Is  known  about  the  category  structure. 
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All  that  must  be  known  Is  that  there  is  a collection  of  observations 
that  are  related  in  some  manner.  Normally,  the  operational  objective 
is  to  discover  a category  structure  idiich  fits  the  observations.  In 
some  situations,  it  Is  possible  to  reduce  a very  large  body  of  data  to 
a relatively  compact  description.  Cluster  analysis  is  also  rejected 
from  consideration.  The  assumptions  that  were  made  before  this  technique 
was  allowed  to  be  considered  were  based  upon  subjective  information. 

The  procedures  used  in  cluster  analysis  to  reduce  the  number  of  variables 
is  contingent  upon  seed  points,  subjectively  established,  and  the  sub- 
jective interpretation  of  the  overall  results  of  the  cluster  anlaysls 
computer  programs.  The  subjective  nature  of  the  cluster  analysis 
techniques  would  compound  the  subjectiveness  already  built  into  the 
methodology  by  these  assumptions. 

Discriminant  analysis  treats  the  problem  of  attempting  to  dif- 
ferentiate between  two  or  more  classes  of  persons  or  objects.  It  attempts 
to  find  a linear  combination  of  variables  such  that  the  distribution  for 
the  classes  or  groups  possess  "little"  overlap.  This  technique  is  ac- 
cepted for  use  because  it  does  address  the  problem  of  two  populations, 
it  does  have  a form  that  permits  the  selection  of  critical  attributes, 
and  it  does  have  a means  by  which  a degree  of  criticality  can  be  assessed. 

The  technique  for  determining  critical  attributes  is  stepwise 
discriminant  analysis.  It  is  used  to  identify  a subset  of  variables 
which  "best"  discriminates  between  populations.  It  is  onl^  necessary  for 
the  purposes  of  this  paper  to  discriminate  between  two  populations,  ac- 
ceptable and  unacceptable.  Therefore,  only  the  special  case  involving 
two  populations  will  be  related  in  this  paper. 
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CHAPTER  IV 

DETERMINING  CRITICAL  ATTRIBUTES  (METHODOLOGY) 

Development 

Stepwise  discriminant  analysis  has  as  its  foundation  one-way 
analysis  of  variance  testing  meians  and  the  discriminant  function.  These 
concepts  are  briefly  explained  in  Appendix  A in  conjunction  with  a 
detailed  explanation  of  stepwise  discriminant  analysis. 

Once  these  underlying  concepts  of  the  multivariate  analysis 
technique  are  understood,  an  overall  methodology  can  be  developed  from 
all  of  the  previous  information.  Stepwise  discriminant  analysis  pro- 
vides a means  by  which  a subset  of  variables  can  be  identified  that 
"best"  discriminates  between  two  populations.  These  two  populations  must 
be  defined  by  individual  mean  vectors  and  a common  covariance  matrix. 

One  sample  of  observations  is  given  in  the  form  of  frequency  distribu- 
tions for  the  attributes.  The  attributes  to  be  considered  must  be  desig- 
nated and  the  mean  and  variance  for  each  attribute  must  be  calculated. 

Since  there  is  not  sufficient  infoxrmation  in  the  original  data 
to  formulate  acceptable  and  unacceptable  mean  vectors  for  the  stepwise 
discriminant  analysis,  these  vectors  will  have  to  be  obtained  subjec- 
tively. There  also  is  not  sufficient  information  from  which  to  formulate 
the  covariance  matrix;  however,  the  sample  variance  of  the  original  data 
is  known.  Hence,  if  the  correlations  between  the  variables  can  be 
estimated,  a covariance  matrix  can  be  formulated. 
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By  generating  multivariate  normal  distributions  utilizing  these 
mean  vectors  and  the  covariance  matrix,  the  stepwise  discriminant  analy- 
sis will  enable  those  attributes  which  "best"  discriminate  between  the 
acceptable  and  unacceptable  populations  to  be  identified.  The  method- 
ology is  best  presented  in  the  form  of  steps  of  a prr endure: 

Step  1:  Examination  and  preparation  of  data 

Step  2:  Determination  of  the  covariance  matrix 

Step  3:  Determination  of  the  mean  vectors 

Step  4:  Generation  of  the  multivariate  observations 

Step  5:  Stepwise  discriminant  analysis 

Step  6:  Analysis  of  results 

Explanation  of  Procedures 
Examination  and  Preparation  of  Data 

The  attributes  of  each  test  must  be  examined  to  determine  which 
level  of  EEA  and  which  sets  of  attributes  are  to  be  examined.  Each 
test  is  different  from  every  other  test.  Even  if  a certain  ttist  is  de- 
signed to  test  a system  that  has  been  previously  evaluated,  there  will 
be  differences  because  of  the  refinements  of  that  first  test.  In  most 
tests,  such  as  FM  286,  the  attributes  to  be  analyzed  would  be  broken  down 
into  sets  of  attributes.  This  would  be  done  because:  (1)  the  number  of 

attributes  is  so  large  that  the  subjective  analysis  of  step  2 would  be 
Incomprehensible,  (2)  the  number  of  attributes  is  so  large  that  the  avail- 
able computers  could  not  handle  the  storage  required  for  the  preparation 
of  data  and  the  stepwise  discriminant  analysis,  and  (3)  the  attribute j 
were  derived  from  the  operational  issues  in  such  a manner  that  natural 
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groupings  of  attributes  would  be  present. 

Having  decided  the  grouping  or  division  of  data,  each  groiq>  aust 
be  examined  to  Insure  that  the  data  for  each  attribute  Is  available  and 
Is  In  the  correct  form.  The  data  for  each  attribute  should  be  repre- 
sented as  a frequency  distribution  with  a range  of  1 to  5 that  coincides 
with  the  rating  categories.  If  an  attribute  does  not  have  a frequency 
distribution  of  observations,  then  It  cannot  be  evaluated.  This  pro- 
cedure does  not  allow  attributes  without  these  data  because  there  Is  no 
means  to  generate  a frequency  distribution  with  the  Information  available 
In  the  test  methodology. 

Once  the  attributes  have  been  organized  In  the  form  of  distribu- 
tions, the  sample  mean  and  .ample  variance  are  calculated.  This  can  be 
accomplished  quite  easily  by  the  use  of  a computer  program  such  as  that 
shown  In  Appendix  B.  This  program  also  tests  the  sets  of  attributes  to 
insure  that  the  assunqitlon  of  multivariate  normality  required  for  step- 
wise discriminant  analysis  Is  met.  In  a discussion  of  multivariate  normal 
distributions,  three  classes  of  distributions  are  of  Interest.  Marginal 
distributions  are  the  univariate  distributions  for  the  Individual  ele- 
ments of  the  vector  variable.  Conditional  distributions  are  the  pre- 
dicted distributions  for  particular  marginal  elements  given  the  known 
distributions  of  the  remainder  of  the  vector  variable.  Conq>onent  dis- 
tributions are  the  distributions  of  any  linear  functions  of  the  vector 
variable. 

If  a vector  variable,  a vector  of  attributes,  has  a multivariate 
normal  distribution,  m.n.d.,  then  every  one  of  its  marginal  distributions 
is  normal.  However  this  is  not  reversible.  If  the  marginals  of  the 
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variables  of  a vector  are  oomally  distributed,  the  vector  is  not 
necessarily  a m.n.d.  (9).  If  a vector  variable  has  a m.n.d.,  then  every 
conditional  distribution  defined  on  it  is  normal  (9).  If  a vector  vari- 
able has  a m.n.d.,  then  each  component  is  normally  distributed.  If  every 
possible  linear  component  of  a vector  variable  is  normally  distributed, 
then  the  vector  variable  has  a m.n.d.  (9). 

There  are  no  universal  goodness  of  fit  tests  for  multivariate 
normal  distributions  (9) . Test  of  multivariate  normality  have  been  pre- 
sented by  Malkovlch  and  Af if 1 (1973) , but  they  are  valid  for  only  a small 
number  of  variables  (33) . Although  the  presence  of  marginal  normality 
does  not  insure  multivariate  normality,  this  methodology  will  test  for 
marginal  normality.  Each  marginal  of  the  vector  is  a special  conqponent 
defined  by  setting  a unit  weight  for  the  assigned  element  and  zero  weights 
for  all  other  elements  (9).  Thus,  it  is  felt  that  a "better"  approxi- 
mation of  multivariate  normality  can  be  obtained  if  the  marginals  are 
normally  distributed. 

In  testing  the  marginals  for  goodness  of  fit,  it  was  discovered 
that  the  Kolmogorov-Smimov  (K-S)  test  is  more  appropriate  than  the  Chi- 
Square  test  for  the  distribution  under  consideration.  The  K-S  test  is 
considered  more  appropriate  because  the  power  of  the  test  is  greater  when 
testing  for  normality  with  p and  estimated  by  x and  s^  (Aflfi  & Azen) 
and  because  Chi-Square  tests  conducted  on  samples  of  data  from  test  FM  286 
showed  this  procedure  infeasible.  In  the  Chi-Square  test  where  the  Chi- 
Square  statistic  is  given  by 
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the  accuracy  of  the  Chi-Square  approximation  improves  as  increases. 

Using  five  as  the  minimal  acceptable  level  of  E^^,  the  number  of  Intervals 

2 

has  to  be  reduced  to  three.  Thus  Xa  k-p-1  zero  where  k equals 

the  nxtmber  of  intervals  and  p equals  the  number  of  estimated  parameters. 

The  hypothesis  that  the  variable  conforms  to  the  hypothesized  density  is 
2 2 

rejected  if  Xq  > Xoj  k-p-1*  always  be  the  case  unless  0^  equals 

for  each  i.  This  occurrence  is  highly  unlikely. 

The  K-S  test  is  nonparametric  and  exact  for  all  sample  sizes. 

In  the  K-S  test,  n observations  are  ordered  from  smallest  to  largest. 
Letting  x^j  de:iv'te  the  i~-  smallest  observation  in  the  sample,  construct 

A 

the  empirical  cumulative  distribution  function  F(x)  defined  by 


F(x)  = 


, X<X(i) 

, i - l,...,n-l 


x>x,  . 
- (n) 


The  test  statistic  D = |f(x)  - F^Cx))  tests  the  null  hypothesis. 

Ho;  F(x)  * F (x) . The  critical  value,  D„,  for  significance  level  is 
0 ^ 

established  from  Table  A6  of  Fishman  (22).  Reject  the  null  hypothesis 
if  D(j<D.  a computer  program  which  futiiltates  making  the  K-S  test  is 
given  in  Appendix  B. 
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If  the  K-S  test  for  normality  shows  that  a frequency  distribution 
for  one  or  more  of  the  attributes  does  not  approximate  a normal  distri- 
bution, then  a transformacion  to  Induce  normality  can  be  utilised.  If 
the  transformed  data  has  been  tested  for  the  normal  fit  and  the  hypoth- 
esis has  failed  to  be  rejected,  the  transformed  data  can  be  utilized  In 
the  remainder  of  the  test.  If  no  transformation  to  Induce  normality  Is 
found,  the  remainder  of  the  procedure  can  be  completed  with  or  without 
this  attribute.  This  point  can  then  be  addressed  In  the  final  analysis. 
Determination  of  the  Covariance  Matrix 

The  determination  of  a covariance  matrix  tluit  approximates  an 
actual  covariance  matrix,  unobtainable  from  actual  data,  Is  an  essential 
step  In  the  process  of  identifying  critical  attributes.  A subjective 
estimate  of  covariance  can  be  formulated  utilizing  the  basic  relation- 
ship of  the  simple  correlation  coefficient  to  the  covariance  of  two 
variables: 


P 


ij 


— 
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If  the  correlation  between  any  two  variables  can  be  determined,  then 
the  individual  variances  of  those  variables  (obtained  in  step  1)  can  be 
utilized.  Thus  the  problem  reduces  to  that  of  estimating  correlation 
coefficients  for  each  pair  of  variables,  or  in  this  case,  attributes. 

The  underlying  assumption  for  stepwise  discriminant  analysis  was 
that  the  vector  of  attributes  under  consideration  has  a multivariate 
normal  distribution.  The  multiple  correlation  coefficient  is  much  more 
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complicated  than  simple  correlation.  For  the  purpose  of  explanation, 
let  Y represent  any  of  the  pfl  attributes  and  Xj^,l>>l,  ...,p  represent  the 
remaining  p attributes.  Also,  let  the  means  and  variances  of  the  attri- 
butes Y ,Xi , . . . ,Xf^,  be  denoted  p , and  o^,  o^, . . . , o^, 

P ylpyl  p 

respectively.  Denote  the  covariance  of  Y and  Xj^  by  o^^.  Correspondingly 
the  simple  con.elatlon  coefficients  will  be  defined  as  given  above. 

These  simple  correlations  do  not  take  into  account  the  presence  of  more 
than  two  attributes.  Since  the  vector  of  attributes  normally  contains 
more  than  two  attributes,  the  correlation  of  Y and  are  actually 
conditional  to  the  values  assumed  by  the  Xj, 

Let  Xj^,...,Xp  be  observations  of  Xj^,.,.,Xp.  There  exists  a 
conditional  distribution  of  Y given  Xj  «xi,  X2  «X2,...,  Xp  »Xp.  In  step 
1 it  was  found  that  if  a vector  of  attributes  has  a m.n.d.,  then  the 
conditional  distributions  are  normally  distributed.  The  conditional 
distribution  of  Y has  mean 


^•xi,...xp  “ ^ + ^i<*r»^i)  +•••+  3p(vV 


This  is  called  the  conditional  expectation  of  Y given  Xj; Xp 

(Afifi  & Azen).  The  quantities  gj^,...,  gp  are  functions  of  the  vari- 
ances and  covariances  of  the  attributes.  This  conditional  distribution 
has  variance 


■=  0^  (1~P?. 
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where  p„  ^ « is  the  multiple  correlation  coefficient  of  T and  X« 

. . . ,X  (Af If 1 & Azen) . Transferring  p„  ^ to  the  left  hand  side 

p y**i».».»*p 

of  the  eqtuitlon  by  simple  algebra  results  In: 


2 2 


Thus,  the  squared  multiple  correlation  coefficient  Is  equal  to  the  pro- 
portion of  the  variance  of  attribute  Y that  Is  "explained"  by  the  linear 
relationship  with  The  multiple  correlation  Is  the  maximum 

simple  correlation  between  attribute  Y and  any  linear  combination  of  the 
remaining  p attributes.  This  multiple  correlation  coefficient  is 
invariant  to  changes  in  scale  (1) . 

The  discussion  of  the  multiple  correlation  coefficient  Illustrates 
that  the  relationship  between  any  two  variables  Is  highly  dependent  upon 
the  remaining  variables  In  the  vector  of  attributes.  The  relationship 
between  Y and  is  dependent  upon  the  "effect"  created  by  the  remaining 
p-1  variables.  With  this  relationship  In  mind,  let  us  continue  to  the 
actual  theory  behind  finding  the  correlation  between  two  variables. 

In  the  case  of  a vector  of  more  than  two  attributes,  multivariate 
normality  was  assumed  if  all  of  the  marginals  were  normal.  Here,  If  two 
marginals  are  normal,  the  assumption  will  be  that  the  Joint  distribution 
is  bivariate  normal.  A second  assumption  will  be  that  the  values  of 
the  remaining  attributes  in  the  vector  can  be  set  or  adjusted  to  any 
appropriate  levels. 

In  a bivariate  normal  distribution,  let  Xj^  and  X2  be  distributed 
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noiaally  with  neans  and  and  variances  and  a^,  respectively.  The 

1 / 1 2 

conditional  distribution  of  %2  given  that  is  univariate  noraal 

(Hines  & Montgoaery)  with  conditional  aean 


012 

W2.1  “ W2  + 


V2 


02 

l*2-l  - + P _ (*1-Pl> 

Ox  (1) 


and  conditional  variance 


0^  - 0|  (X-p2) 


From  equation  (1): 


P2-1  -P2 


P 


(*x-Pi) 

C~^) 


(2) 


The  conditional  mean,  ^2*1*  expected  value  of  X2  give?  the  value 

of  and  Is  the  expected  value  of  Xj^.  The  left  hand  side  of  equa- 
tion (2)  is  the  standard  normal  of  conditional  X2  and  the  expression  in 
the  right  h<.nd  side  is  a constant  times  the  standard  normal  of  Xj^.  Thus, 
the  expected  value  of  X2  given  Xx  is  p times  the  value  of  Xj^  after  nor- 
malizing X2  and  Xj^. 

If  ^2*  °i»  ^2*  P known,  then  for  any  given  Xj^,  the 
value  of  U2«i  determined.  Letting 


42 


O-j.i-i-j) 

( °2  ) 


and  Z, 


(Xj-Hl) 


then 


^2  * ^^1*  Since  and  Z2  if  2^  is  k standard  deviations 

above  its  iaean*  then  the  expect^  value  of  Z2  will  be  p k standard 
deviations  above  its  unconditional  mean.  If  p is  known  and  k varies 
from  the  mean,  then  for  each  value  of  k,  there  is  a corresponding  value 
for  Z2*  If  Z2  is  known,  then  easily  determined.  Con" 

versely,  if  k is  specified  and  Z^  is  estimated,  then  p can  be  calculated. 
By  letting  k vary  and  estimating  corresponding  values  of  Z2>  estimate 
of  p can  be  obtained  by  taking  a simple  average  of  the  p's  corre- 
sponding to  the  k's.  This  is  by  no  means  an  exact  process;  however,  it 
is  a logical  approach  to  estimating  the  correlation  coefficient. 

The  actual  procedure  for  estimating  p approximates  by  x^, 
l>2  by  x^,  (^2^  by  s^^,  and  by  S2  obtained  in  Step  1.  As  mentioned  pre- 
viously, the  relationship  between  any  two  variables  is  contingent  upon 
the  values  of  the  remaining  variables  in  the  vector.  A more  exact  pro- 
cess would  allow  the  "other"  variables  to  assume  all  combinations  of 
values  over  the  entire  range  of  the  rating  categories.  A total  enu- 
meration, in  this  case,  would  be  quite  Infeasible;  consequently,  the 
values  of  calculated  in  Step  1 will  be  considered  as  the  levels  of 
the  "other"  variables. 

The  actual  procedure  for  the  estimation  of  will  consider 
each  pair  of  variables  as  follows: 

1.  Consider  each  of  the  variables  to  be  normalized,  l.e.,  Z-^^'vNCO,  1) 
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and  ZjMI(0,l). 

2.  Let  the  scale  of  neasurement  range  from  1 to  7 corresponding  to  -3 
to  >3  standard  deviations  from  the  mean,  zero. 

3.  For  values  of  ranging  by  integers  from  1 to  7,  excluding  4, 
estimate  the  value  of  Xj  on  the  same  scale  of  1 to  7 where  1 is 
considered  "best",  4 is  considered  “borderline",  and  7 is  considered 
"worst." 

4.  Convert  the  values  obtained  above  from  the  scale  of  1 to  7 to  the 
corresponding  standard  deviation,  -3  to  +3. 

5.  Let  the  values  foe  X^  equal  k.  Let  the  values  for  equal  c. 

For  each  value  of  k,  compute  a corresponding  p.  This  is  calculated 
by  P “ . 


6.  Sun  each  of  the  p's,  and  divide  by  the  total  number  of  estimates,  6. 

7.  Thus,  the  average  of  the  p's  is  the  estimated  p for  and  X.. 

The  estimations  should  be  performed  by  the  Indlvldual(s)  desigi ated  by 
the  agency  chat  is  utilizing  the  overall  methodology. 

Once  all  pairwise  comparisons  are  made  and  all  p^j  arc  estimated, 
the  esimated  covariance  matrix  is  calculated  by  utilizing  the  relation- 
ship of  the  simple  correlation  coefficient  to  the  covariance  of  two 
variables: 


^i 


Letting  be 


and  Oj , then  s , 


the  estimate  of  p^^ 
the  estimate  of 


, Sj  and  s.  be  estimates  of 
is  calculated  as  follows: 
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S 

'Vi 


Sii  Sj^o  ...  Sip 

®21  ®22  * * * ®2p 


®P1 


s 

PP 


where  = s^. 

This  procedure  by  which  Che  covariance  matrix  Is  estimated  has 
limitations  because  of  the  assumptions  that  were  made.  However,  It  does 
reinforce  the  stand  taken  by  OTEA  that  subjective  evaluations  play 
an  extremely  valuable  role  In  Che  design  and  overall  evaluation  of 
operational  tests. 

Determination  of  Mean  Vectors 

The  determination  of  the  acceptable  and  unacceptable  mean  vectors 
to  be  utilized  In  the  stepwise  discriminant  analysis  phase  of  the  method- 
ology to  identify  critical  attributes  is  vital.  The  final  results  hinge 
upon  the  acceptable  and  unacceptable  mean  values  of  the  individual  attri- 
butes. For  this  reason,  it  is  extremely  important  for  those  values  to  be 
determined  by  the  most  knowledgeable  and  capable  individuals  involved 
in  the  operational  tests.  These  individuals  are  involved  with  the 
development  of  the  test  design. 

The  last  step  in  the  development  of  the  Final  Test  Design  is  the 
development  of  analysis  logic.  The  development  of  analysis  logic  is  a 
determination  of  how  the  data  from  the  field  test  will  be  used  to  satisfy 
the  test  objectives.  The  analysis  methodology  includes  how  the  data 
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values  which  were  obtained  In  the  form  of  observations  of  measurable 
data  requirements  are  combined.  Data  values  are  combined  after  con- 
sidering two  types  of  rules,  criteria  and  weighting. 

In  order  to  assess  how  each  aspect  of  a system's  capability 
performs,  a set  of  criteria  for  each  data  requirement  at  each  level  is 
determined.  The  process  of  deciding  the  criteria  is  difficult,  but  it 
can  be  facilitated  by  taking  into  consideration  experience,  values 
derived  from  the  test,  or  comparisons. 

Weighting  is  the  Importance  of  each  data  value  expressed  as  a 
relationship  to  the  other  data  values.  Weighting  takes  into  account 
the  relative  Importance  of  the  data.  Data  values  may  be  given  verbal 
weights  such  as  "essential"  or  "desirable"  or  numerical  weights. 

Those  individuals  that  develop  the  analysis  logic  utilizing  cri- 
teria and  weighting  are,  in  effect,  determining  the  basis  on  which  a 
system  and  its  subsystems  are  considered  acceptable  or  unacceptable.  It 
is  these  individuals  that  should  set  an  acceptable  and  unacceptable  mean 
level  for  each  data  requirement. 

A procedure  for  the  determination  of  the  acceptable  mean  vectors 
is  not  tendered  here.  The  exact  process  by  which  the  criteria  and 
weighting  are  developed  for  the  data  requirements  of  a particular  test 
are  peculiar  to  that  test.  Hence,  the  acceptable  and  unacceptable  mean 
values  for  the  attributes  of  each  vector  are  dependent  upon  the  analysis 
conducted  by  the  test  designers. 

Generation  of  Data 


Once  the  covariance  matrix  and  acceptable  and  unacceptable  mean 
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vectors  have  been  determined.  It  Is  necessary  to  generate  two  sets  of 
multivariate  normal  data  In  order  to  apply  stepwise  discriminant  analysis. 
In  order  to  generate  a multivariate  normal  distribution,  it  is  necessary 
to  use  a fundamental  theorem  of  multivariate  statistical  analysis.  This 
theorem  states  that  if  -Z-  = ...,^  ) is  N(0,I),  then  X with  mean  ji 

and  covariance  matrix  Z can  be  represented  as 

X = C Z + y 

% *\»  'V  ‘Xj 

where  C is  a unique  lower  triangular  matrix  satisfying 

'V 

£ - C C' 

'Xi  “V  'X» 

The  generation  of  X can  now  be  accomplished  by:  (1)  computing  C,  (2) 

*\/  <v 

generating  p Independent  normal  variates,  and  (3)  applying  X » C Z' +y  (22). 

Appendix  A contlans  a computer  program  for  the  generation  of  a multi- 
variate normal  distribution. 

Stepwise  Discriminant  Analysis 

Having  generated  the  necessary  data,  a stepwise  discriminant 
analysis  as  discussed  in  the  methodology  section  is  conducted.  The  F 
to  include,  F to  exclude,  and  the  significance  level  a are  the  means 
by  which  the  degree  of  ciltlcality  can  be  somewhat  controlled.  If  F to 
include  is  set  "too  high",  the  number  of  variables  entering  the  set  of 
attributes  that  discriminates  between  the  populations,  H,  will  be  severely 
restricted.  If  the  F to  exclude  is  set  "too  high",  variables  will  be 
removed  from  the  set  H.  If  the  a level  is  set  "too  low",  then  the 
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F.  level  for  each  step  will  be  "too  high"  and  It  will  affect 

l-a.V1.V2 

entry  Into  H. 

The  F and  a values  mcy  be  adj.^ted  to  fit  the  data.  This  permits 
flexibility  in  the  type  and  quality  of  the  results. 

Analysis  of  Results 

This  final  step  in  the  procedure  to  identify  critical  attributes 
reviews  the  subjective  Inputs,  the  control  parameters,  and  the  final 
results  of  the  stepwise  discriminant  analysis.  These  factors  are  analyzed 
in  an  effort  to  solidify  the  end  results  into  a productive  package  that 
can  be  practically  utilized  in  future  operational  tests. 
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CHAPTER  V 

DEMONSTRATION  OF  THE  METHODOLOGY 
Introduction 

This  chapter  will  demonstrate  the  methodology  presented  in 
Chapter  IV.  The  basis  for  the  demonstration  will  be  the  data  from  the 
Division  Command  Post  Test  FM  286.  Recall  that  the  test  objectives,  to 
evaluate  the  efficiency  of  the  command  post  in  comsand  and  control  of 
division  tactical  operations  and  to  evaluate  the  vulnerability  of  the 
command  post  during  division  tactical  operations,  were  derived  from  oper- 
ational issues.  These  test  objectives  were  further  subdivided  function- 
ally, as  shown  by  Figure  3,  and  operationally,  as  illustrated  by  Figures 
1 and  2.  For  the  purpose  of  clarity,  the  demonstration  will  be  restricted 
to  the  operations  subsystem. 

Conduct  of  the  Methodology 
Examination  and  Preparation  of  Data 

The  operations  subsystem  is  subdivided  into  three  second  level 
essential  elements  of  analysis.  They  Include: 

1.  F.fficiency  in  plans,  orders,  and  report.-^ 

2.  Efficiency  in  providing  Information 

3.  Efficiency  in  supervising  the  execution  of  plans  and  orders 
These  second  level  EEA  are  related;  however,  they  are  still  relatively 
distinct  data  requirements.  At  this  point,  a grouping  of  the  data 
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requlrenents  can  be  initiated.  The  fourth  level  E£A  are  the  measurable 
data  requirements,  or  attributes.  Select  the  efficiency  to  provide  In- 
formation, second  level  EEA,  as  the  example  group.  In  order  to  restrict 
the  size  of  the  problem  for  demonstration,  select  five  variables  for 
conq>arlson.  The  only  restriction  that  is  made  on  the  choice  of  these 
variables  is  that  there  must  be  data  on  these  attributes  in  the  form  of 
frequency  distributions.  The  five  attributes  selected  and  their  fre- 
quencies are  shovra  in  Table  5. 

The  data  on  the  five  attributes  are  examined.  The  sample  mean, 
sanq>le  variance,  sample  standard  deviation,  and  maximum  Kolmogorov- 
Smlmov  statistics  are  calculated.  If  the  max  K-S  statistic  is  less  than 
a critical  level,  , then  the  frequency  distributions  can  be  accepted  as 
normal.  For  an  a-level  of  .05,  D.q^  equals  .886/y^,  where  N is  the 
total  number  of  observations  for  an  attribute.  The  critical  D.q^  levels 
are  as  follows: 


1. 

Attribute  A: 

.0666 

2. 

Attribute  B: 

.0666 

3. 

Attribute  C: 

.0668 

4. 

Attribute  D: 

.0757 

5. 

Attribute  E: 

.0709 

Readily,  it  is  apparent  that  the  original  data  is  rejected  as  being  uni- 
variate normal  for  each  attribute,  see  Table  6.  Tables  7 and  8 also 
show  that  logarithmic  and  square  root  transformations  fall  to  get  the 
data  in  a form  such  that  the  distributions  are  "acceptable"  as  normally 


distributed. 
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Table  6.  Data»  No  Transformatioa 


Variable 

Sample 

Mean 

Sample  Std. 
Deviation 

Sample 

Variance 

Max  K-S 
Statistic 

A 

1 . bXOO 

limtjgyytjjmillll 

U.  533«» 

OTSFIT' 

0.7050 

np^jogj^mniinmiii 

C 

1.9432 

1.4725 

2.1682 

0.2090 

1.9781 

1.2094 

1.4628 

0.4169 

E 

2.5705 

1.2081 

1.4595 

0.2778 

Table  7.  Data,  Log  Transformation 


Variable 

Sample 

Mean 

Sample  Std. 
Deviation 

Sample 

Variance 

Max  K-S 
Statistic 

- A 

0.4135 

B 

0.5151 

0.3995 

0.1596 

0.4917 

C 

0.6571 

0.5376 

0. 2890 

07^7 

D 

0.5283 

0.5480 

0.3003 

0.4153 

£ 

0.8342 

0.4782 

0.2287 

0.2238 

Table  8.  Data,  Square  Root  Transformation 


Variable 


Sample 

Mean 


Sample  Std, 
Deviation 


Sample 

Variance 


0.0710 

0.06>6 

0.5859 

0.1668 

0.1358 


Max  K-S 
Statistic 


■om" 

5.5006 

0.4922 

0.4926 

0.4936 


1.2427 


0.2682 


1.3194 

1.1664 

1.3463 

1.5606 


0.2599 

0.7654 

0.4084 

0.3685 
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There  is  a wide  disparity  between  the  critical  D levels  and  the 
waxlaum  k-S  statistics.  As  shown  by  Figure  5,  the  distribution  "appears" 
to  have  a somewhat  normal  shape.  One  reason  for  the  disparity  is  the 
limited  range  of  observations.  A second  reason  is  that  the  attributes 
are  rated  on  a continuum  from  1 to  5;  however,  the  actual  ratings  are 
restricted  to  integers.  With  these  considerations  in  mind,  the  proce^ 
dure  progresses  to  the  determination  of  the  covariance  matrix. 
Determination  of  the  Covariance  Matrix 

The  first  phase  In  the  determination  of  the  covariance  matrix 
process  Involves  "estimating"  a correlation  coefficient  for  each  pair 
of  attributes. 

Step  1:  Select  a pair  of  attributes  for  consideration. 

Normalize  each  attribute  so  that  the  means  of  each 
attribute  are  equal.  Select  variables  A and  C. 

Step  2;  Let  the  scale  of  measurement  range  from  1 to  7 corre- 
sponding to  -3  to  +3  standard  deviations  from  the 
mean,  zero. 


3 -2 

1 

: -] 

L 0 +1  +: 

1 1 

l +3 

1 i 

1 

1 1 

1 2 3 4 5 6 7 


Step  3:  Let  A vary  from  1 to  7 by  Integers,  consider  all  other 

actrlbutes  to  be  at  their  unnormalized  means,  and 
estimate  C for  every  value  of  A. 


< 


I 


i 

! 


i 

8 S 8 £ 8 SS 


Figure  5. 


Step  4:  Convert  the  values  obtained  from  Step  3 to  standard 


deviations. 


A (k)  C (c) 


-3 

-3 

-2 

-3 

-1 

-3 

+1 

-2 

+2 

-2 

+3 

-1 

Step  5:  Let  the  values  for  A equal  k and  the  values  for 

C equal  c as  shown  above. 

Use  p*  c/j^  and  calculate  the  p 's. 


55 


Step  6:  Sim  the  p's  and  divide  by  6. 

Thus  - 13/36  » 0.3611. 

Step  7:  Finally,  use  the  simple  correlation  coefficient 

to  solve  for  S._. 

AL 

^AC  “ ^AC^A^C  “ (0.7303)  (1.4725) 

- 0.387 

Complete  the  pairwise  comparisons  of  each  pair  and  calculate  the 
"estimated"  covariance;  see  Tables  9 and  10. 

Determination  of  the  Mean  Vector 

The  methodology  calls  for  the  determination  of  the  acceptable 
and  unacceptable  mean  vectors  by  a subjective  analysis  of  the  variables 
involved.  A mean  vector  consisting  of  1.5  for  each  attribute  in  the 
acceptable  population  and  a mean  vector  consisting  of  2.5  for  each  at- 
tribute In  the  unacceptable  population  were  chosen.  In  this  Instance, 
the  values  were  chosen  to  show  what  would  result  from  selecting  a mean 
vector  on  either  side  of  the  sa&\ple  means. 

Generation  of  Data 

Two  sets  of  data  were  generated.  One  set  of  data  had  as  Its 
mean,  the  acceptable  mean  vector.  The  second  set  had  as  Its  mean,  the 
unacceptable  mean  vector.  Both  sets  of  data  used  the  covariance  matrix 
In  the  generation  of  data.  The  printout  of  data  Is  given  In  Appendix  C. 
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Stepwise  Discriminant  Analysis 

Bioaedlcal  Computer  Program  07M  utilized  the  data  generated  in 
the  previous  step.  The  F to  include  was  set  at  0.01  and  the  F to  exclude 
was  set  at  0.005.  The  suosnary  table  is  as  follows: 


Table  11.  Summary 


Step 

Number 

Varlble 

Entered 

F Value 
to  Enter 

Number  of 

Variables  Included 

U Sta- 
tistic 

1 

! 

2 

111.2828 

1 

0.6402 

2 

1 

14.0945 

2 

0.5974 

3 

5 

1 

5.8616 

3 

0.5801 

4 

^ 1 

3.4448 

4 

0.5700 

5 i 

3 

1.4660 

5 

0.5688 

The  results  of  each  step  in  the  program  are  given  in  Appendix  C. 
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Using  an  a level  of  0.05»  the  summary  table  is  now  used  to  find 
which  attributes  should  be  used  in  the  "best"  classification  procedure. 


F Statistic 

“ ^'95,1,196  " 

^l-a.v^.v^  “ ^*93.1,197  " 3-9 

~ 1 1 QA  ~ 3.9 

l~a»v^»V2  95,1,196 

F.  — F.qc  1 -jAc  ~ 3.9 

l~a>Vj»V2  95,1,195 

1— a,v^,y^  95,1,194 


F to  enter 
111.2828 
14.0945 
5.8616 
3.4448 
1.4660 


Variable 

2 

1 

5 

4 

3 


Since  the  F to  enter  for  variable  4 is  less  than  F*g5  2.95 
H {2,1,5},  or  in  the  notation  used  in  finding  the  covariance  matrix 
H = {B,A,E}.  Now,  since  the  third  step  was  the  last  step  in  which  a vari- 
able entered,  formulate  the  linear  discriminant  function  from  the  co- 
efficients and  constants  at  Step  3. 


^i 

“ ®li 

- a2i 

a 

“ a 

-a  = 

1.91947  - 3.1652  - 1.24615 

1 

11 

21 

a 

» a 

- a = 

1.67710  - 2.83173  - -1.15463 

2 

12 

22 

"5 

‘*15 

- a “ 
25 

.65847  - 1.11464  - -.45617 

c 

■ c - 

/I 

■ c ■ - 
1 

9.57256  - (-3.87143)  = -5.70113 

Thus,  classify  x into  W , the  acceptable  population,  if 

"V/  1 

Z = -1.24615X  - 1.15463X  - .4567x  > - 5.70113 

1 2 3 - 

where  the  prior  probabilities  are  equal. 
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Now  calculate  the  estimated  Mahalanobis  distance, 

<1 

for  each  step  q. 


2 q(n  -hi  )(n  +n  -2) 

D “ 12  12  F 

q n n (n  -hi  -q-1) 

12  12 


where  F Is  the  approximation  to  the  U statistic.  F Is  an  exact  approx- 
imation In  this  case  because  there  are  two  populations. 


2 1(200) (198) 

D ■ 

1 (100) (100) (198) 

2 2(200) (198) 

°2  “ (100) (100) (197) 

2 3(200) (198) 

* (100) (100) (196) 

2 A (200) (198) 

* (100) (100) (195) 

2 5 (200) (198) 

°5  “ (100) (100) (194) 


(0.6402)  > 0.012804 


(05.5974)  - 0.040203 


(0.5801)  - 0.0606122 


(0.5700)  - 0.0812307 


(0.5628)  - 0.1020618 


is  equivalent  to  testing  to  see  If  the 

q P 

last  attributes  contribute  to  the  discrimination  achieved  by  the 

2 2 

attributes  In  H.  Approximate  A by  D , where  m»p  or  q.  Testing 

m m 


Ho  : D3  - dJ.  q-3,  p-4 
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F 


(n  +n  -p-1) 
12 

(p-q) 


2 2 
n n^(D  -D  ) 

.J  2 p..  q 

2 

(n  +n  ) (n  +n  -2)+n  n D 
12  12  1 2 q 


F - 1.005127 


F - F =3.9 

l-a,p-q,n  +n  -p-1  .95,1,195 

1 2 


F > F,  hence  "fail  to  reject"  the  hypothesis. 

.95,1,195 


Testing 


2 2 
H : D - D 
0 3 5 


F - 1.010309 


F 

1-0,2,194 


4.79 


F > F,  hence,  "fail  to  reject"  the  hypothesis. 

.95,2,194 


Analysis  of  Results 

The  results  of  the  stepwise  discriminant  procedure  showed  that 
variables  B,  A,  and  E "best"  discriminated  between  the  acceptable  and 
unacceptable  populations.  The  test  of  the  estimated  Mahaltmobls  distances 
showed  that  the  discrimination  was  significant  at  the  .05  level.  These 
findings  reinforce  the  findings  of  the  stepwise  process.  Although  some 
authors,  i.e.,  Cooley  and  Lohnes,  point  out  that  marginal  normality  need 
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not  be  tested  when  making  an  as8usq>tion  of  multivariate  normality,  the 
results  of  the  K-S  test  shewed  that  the  distribution  of  the  saiq>llng  data 
was  not  univariate  normal  for  any  attribute.  If,  In  fact,  the  asstimptijn 
of  multivariate  normality  Is  violated,  then  the  validity  of  the  results 
is  questionable.  If  the  subjective  development  of  the  covariance  matrix 
Is  not  consistent,  then  the  resulting  simulated  data  is  not  truly  repre- 
sentative of  the  populations. 

There  are  Inadequacies  In  the  process,  but  the  approximation  or 
estimated  results  are  still  useful  for  the  purpose  for  which  the  process 
was  developed.  That  purpose  Is  to  provide  an  aid  by  which  a test  de- 
signer could  determine  which  attributes  from  a group  of  attributes  "best" 
distinguish  between  an  acceptable  and  an  unacceptable  system. 
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CHAPTER  VI 

COMCLUSIUNS  AND  REC(»IMENDAIIONS 
Conclusions 

Multlvsrlste  analysis  techniques  may  be  a valuable  aid  In  deter- 
minings operatlonalys  which  attributes  are  more  useful  In  dlstlngulshl-'g 
between  an  acceptable  and  an  unacceptable  system  or  subsystem.  The  ' se 
of  multivariate  analysis.  In  general,  and  discriminant  analysis,  in  par- 
ticular, are  adaptable  to  "real  world"  operational  tests.  However,  oper- 
ational tests  and  force  development  testing  a'nd  experimentation  are  not 
structurally  designed  to  facilitate  the  p.ppllcatlon  of  multivariate 
statistical  procedures. 

Given  the  present  test  structure,  a viable  approach  to  deter- 
mining critical  attributes  Is  to  Incorporate  a subjectively  determined 
covariance  matrix,  subjectively  determined  acceptable  and  unacceptable 
mean  vectors,  and  stepwise  discriminant  analysis  in  a methodology.  Once 
the  covariance  matrix  has  been  determined,  the  selection  of  the  mean 
attribute  vectors  can  be  varied  to  solicit  different  results  from  the 
stepwise  discriminant  analysis.  It  appears  that  the  closer  the  accept- 
able and  unacceptable  vectors  bracket  the  sample  mean,  the  more  sensitive 
the  process  Is  to  critical  attribute  selection. 

Limitations 

This  research  was  conducted  under  the  premise  that  the  basic  test 
design  was  not  going  to  be  changed.  Consequently,  It  Is  limited  by  the 
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assuaptlon  of  aarginal  noxaality  for  the  observations  on  each  attribute 
(and  thus  oniltlvarlate  noraallty).  This  thesis  did  not  address  the 
feasibility  of  altering  the  basic  test  design  In  order  to  Insure  the 
validity  of  the  noraallty  assurq>tlon. 

The  data  were  analyzed  with  the  sole  purpose  of  deterolnlng  the 
critical  attributes.  The  applicability  of  nultlvarlate  analysis  tech- 
niques In  the  evaluation  of  the  effectiveness  of  the  covaand  and  control 
system  was  not  considered. 

The  data  were  concerned  with  discrete  observations  over  a rela- 
tively small  range.  The  multivariate  statistical  theory  Is  founded  on 
the  assumption  of  a continuous  distribution  of  observations  for  each 
attribute. 

This  research  is  further  limited  by  the  degree  to  which  Individual 
and  staff  test  designers  are  able  to  subjectively  evaluate  measures  of 
effectiveness  and  to  subjectively  establish  which  mean  values  contribute 
to  a successful  and  unsuccessful  system.  Additionally,  the  definition 
of  the  term  "critical  attribute"  precludes  the  use  of  other  multivariate 
analysis  techniques. 


Recoamendatlons 

There  should  be  an  emphasis  put  on  test  design  that  would  permit 
the  use  of  multivariate  analysis  techniques  to  be  more  applicable.  The 
applicability  of  nonparametrlc  statistics  in  determining  the  correlations 
among  variables  should  be  Investigated.  The  data  collection  procedure 
could  be  designed  to  permit  the  use  of  serial  correlation.  It  could 
also  be  modified  to  enhance  the  normality  of  the  distribution  of 
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observations  for  each  attribute.  A more  flexible  rating  scale  should 
be  developed  that  would  enhance  critical  attribute  detection.  In  order 
to  insure  the  validity  of  the  normality  assuiq>tlon,  the  rating  scheme 
could  be  Improved  by  extending  the  range  of  the  scale  or  by  converting 
to  some  ocher  continuous  rating  scale. 

The  methodology  developed  in  this  thesis  should  be  lnq>lemented  in 
future  operational  tests.  This  application  would  enhance  the  validity 
of  this  technique,  as  well  as  other  multivariate  techniques,  and  assist 
In  the  design  of  forthcoming  operational  tests. 

Further  study  needs  to  be  done  In  the  area  of  the  sensitivity  of 
stepwise  discrimination  to  the  mean  vectors.  A guideline  for  the  selec- 
tion of  the  mean  vectors  to  facilitate  the  degree  to  which  critical 
attributes  are  selected  would  be  highly  useful  In  future  applications  of 
this  methodology. 
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APPENDIX  A 


Pretediig  pa|9  Hank 
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ONE-WAY  ANOVA 

One-way  ANOVA  Is  basically  a statistical  procedure  for  testing 
the  equality  of  several  means.  The  underlying  theory  Is  founded  upon 
a linear  statistical  model  by  which  a number  of  populations  are  compared 
on  the  basis  of  observations  on  one  random  variable.  A detailed  discus- 
sion of  the  theory  Is  available  In  Hines  and  Montgomery  (27).  In 
general,  the  total  variation  In  the  data  is  partitioned  into  component 
parts.  These  component  parts,  differences  between  populations  and 
differences  within  populations,  are  used  to  develop  a test  statistic. 

SS^  - SSg  + ss„ 

SS  Is  the  total  sum  of  squares;  SS  is  the  between-populatlon  sum  of 

squares;  and  SS^  Is  the  wlthln-populatlon  sum  of  squares.  Let  a be  the 

number  of  populations  and  N be  the  total  number  of  observations  on  a 

random  variable,  X.  Let  the  null  hypothesis  be  that  the  means  of  the 

populations  are  equal.  It  Is  assumed  that  X Is  normally  distributed; 

thus,  SS^/o  Is  a Chi-Square  distribution  with  N-1  degrees  of  freedom,  df. 

Hence,  SS. ,/a^  is  y^(N-a)  and,  if  the  means  are  equal,  SSp/a^  is  (a-1). 
W ° 

Under  the  null  hypothesis,  H^, 

SS„  SS  SS„ 

T , B W 

~2~  ~2~  ~Y~ 

X^i.N-1)  » x^(a-l)  + X^(N-a) 


or 


It  la 


2 2 2 
vhere  SS^/o  and  SS^/o  are  Independent  x randoa  varlablea. 

alao  known  that  - SSg/\a-l)  la  F(a-l,M>a).  Call  SS^/Ca-l) 

SSy/N>a) 

and  SS^/(Vha)  mean  aquarea,  MS^  and  MSy  respectively.  It  can  be  ahoim 
that  Fq  la  an  appropriate  test  statistic  by  taking  the  ei^ected  value 
of  MSg  and  MS^.  If  the  value  of  F^  Is  too  large,  then  the  null 
hypothesis  that  the  means  are  equal  should  be  rejected.  It  will  be  seen 
In  the  stepwise  discriminant  analysis  procedure  that  It  Is  highly 
desirable  to  reject  the  null  hypothesis  and  to  have  the  highest  value 
of  Fq  that  Is  possible. 


DISCRIMINANT  FUNCTION 

The  standard  classification  procedure  for  p continuous  variables 

assumes  that  a vector  of  observations  comes  from  one  of  two  multivariate 

normal  populations.  Let  the  vector  of  observations  be  represented  by 

X ■ (x. , ...,  x-)”*  and  assume  that  one  population,  W,  Is  N(y?*^,  ZP*P) 

'V,  * P A 

and  the  second  ,/opulation,  W_,  is  N(yP^^,  EP*P). 

I <\,2  'V. 

If  )ii,  yo>  ^ assumed  to  be  known,  then  It  seems  reasonable, 

intuitively,  that  a linear  combination  of  the  observations  can  be  found 
by  which  that  vector  of  observations  can  be  classified  Into  W^,  or  W2. 

The  linear  combination  of  observations 

h “ “1*1  “2*2  ®p*p 

is  '.ailed  a discriminant  function.  Classify  x into  W.  if  Z is  greater 
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than  or  equal  to  soae  constant  C or  classify  x into  Uo  if  Z is  less  than 
that  constant.  Thus,  if  the  and  C can  be  detemined  such  that  the 
probability  of  naking  an  incorrect  classification  is  miniaized,  then  the 
problem  is  solved. 

Suppose  X is  front  W. . In  this  case,  Z is  N(c, , o^)  where 

A-  X X z 


- £ “j^lj* 


2 **  P 

J = E E 0.0. .a., 
^ 1-1  j-i  ^ ^ 


I£  X Is  from  Chen  Z Is  N(f2.  opuhere 


Co  » E and 

j»l  ^ 


Is  given  above.  In  order  to  maximize  the  distance  between  the  two 
populations,  the  o^^  should  be  chosen  so  that  the  means  of  the  two  popu- 
lations are  as  far  apart  as  possible.  Thus  the  Mahalanobis  distance 


can  be  utilized.  It  can  be  shown  that  the  o^  coefficients  which  maximize 

2 

A are  the  solutions  to  the  set  of  linear  equations 


OsOii  + OoCio  +. . .+  0 


pOip  - Jiii-P21 
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Cl  ^0 2^  ^ ^2^22  ^ 

Vpi  “2®P2 

The  dlscrlalnent  score  Z for  a vector 
using  the  obtained  from  the  solution  of  (3)  in  equation  (1). 

Intuitively,  the  constant  c would  be  that  point  between  and 
that  minimized  the  probability  of  classifying  x into  W. , or  Wo  incorrectly. 

J.  * 

Since  the  variance  for  both  populations  are  equal,  then  it  seems  obvious 
that  C should  be  the  midpoint  between  snd  Figure  6). 

Thus  the  procedure  is  to  classify  x into  W,  if  Z>C  or 

A.  i — 


OpOzp  - M12-M22 

VpP  “ »‘lP’»'2p 

of  observations  can  be  found  by 


P 

E a.x.  > 
j-1  J J " 


and  to  classify  x onto  W»  if  Z<C  or 


P 

E OjXj  < ^^+<^2 
j-1  2 . 

It  can  be  shoim  (1)  that  this  intuitive  approach  is  correct  if  the  a pri- 
ori probability  that  a vector  comes  from  is  equal  to  the  a priori  pro- 
bability that  it  comes  from  W2  and  if  the  costs  of  misclassif ication  are 
equal.  Otherwise,  from  the  generalized  Bayes  classification  procedure 
presented  by  Afifi  and  Azen  classify  x into  W.  if 
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2 qj  C(2|l) 


and  classify  x into  U2  if 


Cl+C2  q2  C(l|2) 

Z < + In 

2 q^  C(2|l) 


where  q^  are  the  a priori  probabilities  for  being  classified  into  popula- 
tion 1*1,2  and  C(2(l)  and  C(l|2)  are  the  costs  of  mlsclasslflcation. 

It  was  assumed  initially  that  the  parameters  of  the  population 
distributions  were  knoim.  If  unknown  and  if 

1*1,2  are  independent  random  samples  from  and  W2,  then  % 

can  be  estimated  by  x,,Xo,  and  S,  respectively,  where  x. , 1*1,2  are  sample 
means  and  S is  the  pooled  sample  covariance  matrix.  These  consistent 
estimators  are  applied  to  the  generalized  Bayes  classification  procedure 
which  becomes  an  estimated  generalized  Bayes  classification  procedure. 
Using  x^,  1*1,2,  j*l,...,p  and  Sj„,,m*l, . . . ,p  in  equation  (3),  solve  for 
estimates  of  denoted  by  a^.  Use  the  a^  to  calculate  the  estimated 
discriminant  score  for  each  observation  vector  k*l,...,nj_.  Esti- 


_ 1 "1 

"1  ■ 71  Li 


and  estimate  by 


= 2;  E a,  S4„  a_, 

j=l  m*l 


73 


Thus  the  estimated  generalized  procedure  modifies  equations  (4)  and  (5) 
so  that  X is  classified  into  if 


p Zj^+Z2  q£  C(lj2) 

Z « Z a .X , ^ + In 

1*1  ? C(2|l)  (6) 


and  X Is  classified  Into  if 

A.  ^ 


Z^+Z,  q,  c(l|2) 

Z < + In  

2 q^  C(2|l) 

2 

The  estimate  of  A Is  the  sample  Mahalanobls  distance^ 


2 (Zi-Z2)2 

D • 

2 

®Z 

Under  the  assumptions  that  the  several  original  variables  have  a multi- 
variate normal  distribution  within  the  populations  from  whlc  . the  samples 

were  drawn  and  that  the  covariance  matrices  for  the  two  populations  are 

2 2 

equal,  an  F statistic  derived  from  D can  be  used  to  test  L *0  or 
equivalently 


nj^n2  (nj^+n2-p-l) 

F = 

p(nj^+n2)(n2^+n2”^) 


D 


2 


F is  compared  with  Fi-ci,p,ni+n2-p-l.  Thus,  this  F statistic  is  the  same 
as  that  F which  utilizes  the  two-sample  Hotelling  statistic  for  testing 


equality  of  mean  vectors. 
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nj^4«2-p-l  2 

F « T 

(nj^+n2-2)p 

STEPWISE  DISCRIMINANT  ANALYSIS 

Stepwise  discriminant  analysis  is  a multivariate  analysis  technique 
by  which  a subset  of  variables  that  "best"  discriminate  between  k popula- 
tions can  be  identified  from  the  whole  set  of  variables.  This  thesis  is 
only  concerned  with  two  populations.  The  case  of  k»2  will  be  discussed 
here.  The  procedures  for  the  general  case,  where  k can  represent  multi- 
ple populations,  is  presented  in  Aflfi  and  Azen  (1).  The  terminology  used 
by  Afifl  and  Azen  is  adapted  here. 

Consider  two  multivariate  normal  populations,  and  W2,  with 
parameters  discussed  above  where  a p dimensional  observation  vector  was 
classified  into  one  of  two  imiltivaTlate  normal  populations.  Hie  general 
logic  behind  the  stepwise  discriminant  procedure  is  to  first  Identify  the 
variable  for  which  the  mean  values  in  the  two  populations  are  "most  dif- 
ferent." This  variable  will  be  entered  into  a separate  set.  Thereafter, 
on  successive  steps,  the  conditional  distribution  of  each  variable  not 
entered,  given  the  variables  entered,  will  be  considered.  Of  the  vari- 
ables not  entered,  the  variable  for  each  mean  value  of  the  conditional 
distributions  in  the  two  populations  which  are  "most  different"  will  be 
identified.  This  variable  will  be  added  to  the  separate  see.  ''he  step- 
wise process  is  stopped  when  no  additional  variables  significantly  con- 
tribute to  the  discrimination  between  the  two  populations.  The  measure 
by  which  the  "most  different"  variables  are  selected  is  a one-way  analysis 
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of  variance  (AHOVA)  F statistic. 

A one-way  ANOVA  F statistic  that  is  calculated  on  those  variables 
not  entered  into  the  separate  set  is  called  an  F to  enter.  A one-way  ANOVA 
F statistic  calculated  for  those  variables  that  are  in  the  separate  set  is 
called  an  F to  remove.  There  are  two  fixed  F statistics  that  are  used  as 
control  paraaeters  in  the  procedure.  These  are  F to  include  and  F to 
exclude.  These  are  minimal  acceptable  values  for  F to  enter  and  F to 
remove. 

A detailed  program  for  the  stepwise  procedure  is  as  follows: 

Step  0:  Let  . . .,xP’‘^,  and  . . . ,xP*^  be  random  samples  from  Wi 

11  Ini  21  2n2 

and  W^,  respectively.  The  F to  enter  along  with  its  degree  of 
freedom  (df)  is  computed  for  each  Xj , j«l, . . . ,p.  This  F is  a one- 
way ANOVA  F statistic  for  testing  Hq;  uij*P2j»  j*l» •••»?• 

If  all  F to  enter  are  less  than  F to  include,  a prescribed  in- 
clusion level,  the  process  is  terminated.  If  this  occurs,  the 
conclusion  Is  that  no  variable  significantly  discriminates 
between  the  two  populations. 

Step  1:  The  variable  X4  having  the  largest  F to  enter  Is  chosen  as  the 

•^1 

first  variable  to  enter  the  separate  set,  H.  The  estimated  lin- 
ear discriminant  coefficient  and  constant  are  calculated  for  both 
popu3.atic;n  W,  and  Wo.  The  classification  table,  U statistic, 
and  &v.  F appiToximacion  'cj  U are  calculated..  The  F to  remove  for 

Xs  , which  is  etjvjal  lo  the  F to  enter,  and  its  df  is  calculated. 

'1 

The  F to  enter  and  its  df  for  each  variable  not  entered  are 
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calculated.  These  test  the  hypothesis  Pjj,j  *y2j  j vhere 
j is  the  mean  of  the  conditional  distribution  in  of  Xj 
given  Xj^,  i*l,2,  j»l,...,p,  If  all  the  F to  enter  are 

less  than  F to  Include,  then  the  last  step,  Step  S,  is  executed. 
Otherwise,  Step  2 is  executed. 

Step  2:  llie  variable  X«  is  chosen  from  those  F to  enter  computed  in 

•'2 

step  1 that  has  the  maximum  F.  Thus,  now  the  separate  set  con- 
tains X^  and  X.  , H » {X^  X^  ).  The  two  estimated  linear 

Jl  2 2 

discriminant  coefficients  and  constant  are  calculated  for  each 

population  and  W2.  The  classified  table,  U statistic,  and  F 

approximation  to  U are  calculated.  The  F to  remove  and  their  df 

are  calculated  for  and  X*  . These  test  Hj.:  y,.  »U-. 

Jl  J2  ^Jl  J2  ^1*^2 

and  Hn:  u, , , -mo . , , where  u..  . . is  the  nu  an  of  the  con- 

■'2-V 

ditional  distribution  in  of  Xj  given  Xj^  and  Xj^,  i*l,2, 
j-l,...,p,  or  j2* 

Step  3:  (a)  Letting  L denote  the  set  of  k variables  which  have  been 

entered  (replacing  H).  If  any  of  the  F to  remove  for  the  vari- 
ables In  L are  less  than  F to  exclude,  a prescribed  deletion 
level,  delete  from  L the  variable  with  the  smallest  F to  remove 
and  execute  (b)  with  k-1  replacing  k.  If  all  the  F to  enter  for 
the  variables  not  in  L are  less  chan  F to  include,  execute  Step 
S.  Otherwise,  choose  the  variable  with  the  largest  F to  enter, 
and  add  it  to  L so  that  k***!  replaces  k. 

(b)  The  k estimated  linear  coefficients  and  constant  are  cal- 
culated for  and  W2.  The  classification  table,  U statistic, 
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and  F approxloation  to  U are  calculated.  Calculate  the  F to 
renove  and  its  d£  for  each  variable  in  L.  These  test 
"‘'2s.(k-D  X in  L given  the  reaaining  k-1  variables  in 

L.  The  notation  Mis. (k-1)  is  the  mean  of  the  conditional  dis- 
tribution in  Vi  of  X3  given  all  the  other  variables  in  L except 
X3.  The  F to  enter  and  their  df  are  calculated  for  those  vari- 
ables not  in  L.  These  test  Hq:  Pij  .k>  where  Uij.k  is  the 

mean  of  the  conditional  distribution  in  Uj^  of  Xj  given  all  the 
variables  in  L where  1*1,2,  j«l, ...,p,Xj  not  in  L. 

Step  4,5,...:  Repeat  Step  3 recursively  until  all  the  variables  have 

been  entered  and  no  F to  remove  is  less  than  the  F to  exclude  or 
when  the  F to  enter  is  less  than  the  F to  include  for  all  vari- 
ables not  in  L.  Then  execute  Step  S. 

Step  S:  The  posterior  probability  of  belonging  to  and  W2  is  calculated 

for  and  X2k  where  k designates  the  variables  entered.  These 
probabilities  are  used  to  classify  each  set  of  data  into  one  of 
the  two  populations  and  a classification  table  is  prepared.  A 
S’uonary  table  is  prepared.  In  this  table,  the  step  number,  the 
variable  entered  or  removed,  :he  F to  enter  or  F to  remove,  the 
U statistic,  and  the  F approximation  tc  U is  given  for  each  step. 

This  completes  the  stepwise  discriminant  analysis  procedure.  The 
summary  table  is  then  used  to  determine  which  variables  "best"  discrim- 
inate between  the  two  populations.  Let  a be  a prescribed  significance 
level.  If  the  F to  enter  for  variable  Xj^  in  Step  1 is  less  tluin 

V V,  then  it  is  not  possible  to  significantly  discriminate  between  the 

1 » z • 
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two  populations.  Otherwise,  H « {X.  }.  In  Step  2,  X.  was  entered.  If 

J2 

its  F to  enter  is  lesfi  than  ^ Oth«twlse,  H ■ 

{X  ,X,„}.  For  each  step  thereafter,  if  a variable  is  rcaoved,  delete  it 
J1  J2 

frow  H and  go  on  to  the  next  step.  If  a variable  is  entered,  caviare  its 

F to  enter  with  F,  . If  its  F to  enter  is  less  than  F,  , 

l-o,Vj^,V2  1h»,v^,V2 

stop  aa«jnding  U.  Otherwise,  augment  H and  go  on  to  the  next  step. 

Once  all  variables  that  are  eligible  for  entry  have  entered  H,  the 
linear  discriminant  function,  d,  can  be  obtained  by  taking  the  difference 
of  the  two  estimated  discriminant  scores. 


* *li  “ ®2i 

C - 5^  - C2  - Cl 
2 

d » I ^I’^i 
x^eH 

Thus,  an  observation  vector  containing  those  variables  in  H can  be  classi- 
fied into  population  if  d^C.  Let  q designate  the  variables  in  H. 

Since  there  are  only  two  populations,  the  F approximation  to  U is  exact. 

The  estimated  Mahalanobls  distance  based  on  q variables  may  be  obtained 
from  the  F approximation  to  U. 

q(n,+n2)(ni+n2-2) 

D/  . 

H nj^n2(ni+n2-q-ll 

for  q«l,...,p  and  the  sample  sizes  and  n2  from  Wj^  and  W2,  respectively. 


Suppose  the  variables  X,  X_  are  in  H.  To  test  that  the  remaining 
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X^l  Xp  do  not  contribute  to  the  discriaination  achieved  by  the  varl- 

ablea  in  H.  test  H : A*  ■>  This  tests  the  difference  between  the 
o q p 

population  Mahalanobis  distance  based  on  the  q variables  in  H and  all  p 
variables.  Let  and  be  the  sample  estimates  of  A^  and  A^.  Then, 


2 2 

ni+n2-p-l  nin2^°p“®q) 

p-q  (n^+n2)  (nj^+n2~2)  + nin2®q 

Under  Hq,  this  F has  a F „ _ + , distribution.  If  F>F, 

° p-q.n^+n2-p-l  1-0, p-q, 

ni+n2“P"l*  *^*J®*^^  ^o*  *^o  rejected,  this  «>ould  mean  that  one  or 

more  of  the  p-q  variables  not  in  H,  contribute  to  the  discrimination 
between  and  W2  for  a given  significance  level. 
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~1T11464 

cowsTsm" 

-3,87143 

-9.57256 

Number  of 

CASES  classified  INTO  GROUP  . 

Group 

Arc— 

WACC 

KO — 

UNfccC 


20 


8o 


SfEP  number 

— VAT<lAai.r  ENTEREP •» 

“V;;i^IABLE^rtWCi;TP*^*MtTTn'(rREH?i«Er*-"OE6REtS  Or-|rREtDOM X'  195  ~ 

I"l6,0225 2 576¥8S 4 3i4448  5 8^6220  " 

"~V7KIAgLeS"WOnWtLOOEO  -ANJ  >-7(T  EMTeft--~OE«WEtS"OF~PftEEOoW 1 — 194 

3-  -y;in56(r : r 

■" TJiSTAflSTIC : 757073 DEGREES  O'"  FKEEqoM  ~ l " 198 

A»>PH0x1MATE  F 36.77213  OEG^EEs  0>-  FKEEqqM IL-i?5*jy! 

f MATi^IX  - degrees  Of  freedom  4 t95 _ _ . _ 

group 



GhOUP 

OrrACc 38.77^12 

mncriON 

ACC  UNACC 

~Y4PrABcr 

1 2.46923  4.21784 

2 1 ,3gam  2.31706 

4 -.73664  -1.36077 

5 r.-230V4 2718663 

“■C(5NSTAf.T _ 

-3.95552  -9.85936 


number  op  cases  clascIFIEq  ii|TO  group  - 

ACC"UMjCC 

Group 

— AcC 81 19 

UNAcC  22  7a 
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"•SlEP~iJOHBER 9 

Variable  entered  3 

Variables  iNcuuuta  a^o  f to  rehove  - degrees  of  freedom 

1 194 

I 17.7206  2 H.2594 

3 1*4660 

4 4.8S32 

5 B.2265 

U-STAtXSTIC  .56675 

OEOoEEs  OE  freedom  5 

1 198 

ApPROxTWATET  j^.78l2i 

□EOrEE^;  qF  freedom  3 

19«.00 

Ff’KT r1  5T  - degrees  Op  h'REEQoM 

~5 

5R0W 

ACC 


UrOUP 

UNACc 

29.78120 

F level 

INSUFFICIENT  FOR  FURTHEb  COMpUTATlON 

function 

variable 

ACC 

UNACC 

1 • 
2 

“2,p8i34 

1.26781 

“479S263  ■ ~ 

2.10980 

"*3i9w6 

-1.13178 

"*59794  • 

-2*10086 

“5 

T;745V8 

T*.  13679  ■ - - - 

Cor  STA^T 

-3.98969 

-9.90005 

wKOUP  WITH  Square  of  OiSTo.'JCC  FHqM  aNU  posterior 

' C,.hGEsT“PR'6ffJ  probability  FOR  grOUr  - 


"oROUp  ACc UfiACC- 

ACC 


TA5E 

1 

ACC 

7.703 

.662. 

9»n50 

,338» 

■ 2 ~ 

•ACC 

8, 057- 

. 960  * 

14.426 

,040* 

3 

ACC 

4,497 

.7l0> 

6.385 

.290» 

• 4- 

ACC- 

-6,164- 

'.676'. 10ib84 

-71247 — 

5 

ACC 

7,137 

,943* 

17.0I8 

,007. 

b 

■ ACC~ 

T2',TJin}" 

• 6 ^3. 

— 15.355— 

“,727» 

7 

UUACC 

10,377 

.2?3. 

7.R76 

,777. 

6 

- ACC 

•9.031  - 

.900. 

I7.3Q2" 

-.012.  - 

9 

ACC 

7.705 

.934. 

I3.c21 

,066* 

~io 

ACC 37435“' 

",7597“ 

- 5.732 

,241*  — 

11 

ACC 

2,727 

.539. 

3.  >99 

,441. 

12 

AcC 

3,553 

.5437- 

71577 

13 

UllACC 

6,237 

,271, 

4,359 

.729* 

■ 14 

r;876“- 

.■617." 

10.0&8 

,183* 

15 

ACC 

8.670 

.963. 

15. >72 

,037* 

• J6 

ACC 2.E83" 

.906, 

7.120 

,094*— 

17 

ACC 

5,209 

.622. 

6.369 

,378. 

16 

'ACC' 

7 “1 60 

T991, 

■,0097 

• 19 

ACC 

2,996 

.951, 

8.932 

,049. 

— 20 

ACC- 

-7,233- 

IBOS.  ' 

- 7.369 

,495*-  - 

■«005?~ 

.088* 


10.673 

6.s31 

~4;a8o 
12«q92 
1275' 
9.^59 


.037* 


;o92 — T309T— 
6*636__.22S* 
-4,i27-  .awr 
12.228  .093»_ 
■97205  ’."TTST 
7.c9? 

Tr.076— .071* 
7.. ,98  .998* 

”4.594  ,731* 

8.o87 

37o22-".‘752“ 


.268»~  '"'■‘77  '■^OR. 

iSP.Sf r*  • 24^f — 

-.*■..  .7511  *u„, 

7.913  .94A*_  

■77995-7805.  •JJIT 

5,120  .992._5i056_i0® 

r;95T--7f2r  ‘Siy, 

77543 — 7905*  i2TQ4l  7^ 


6,731  .560. 

■67876  77967 

5.2«1  .273. 

77253— 76967 
10,588  ,655. 


7,^14  .940J 

^;^98--204. 

3.3I8  ,727J 

xrrij;95  P?57~ 


14.i44  ,X95* 

675S^-.9^sr  -i9-.l6a- 


U-jACC  9,793__,235j. 
U'lACC — 1x7708  ,292. 

ACC  h:172  .977. 

7.953 — 7J?5ff7 


2.432.  ,^65»  . 
-9.996'  .^^8* 
u.a27 _4«;_ 
'5risr-7l^* 


6,436  .796i  9.h55  .204 

-«r,04T:  795'8.  *?R?, 

5.114  ,846. -*olr»  ~ 

5.662~','9337  *??!, 

3.908  ,890. 

6-.  348  VETBl 

2.651  .957. 

^,'486.  ^*3®' 


ACC 

WACC 67494 

ACC  8,885  .512. 

■*CC 4^;2CT4  ;?95. 

ACC  8,966  ,989. 

77x35 


7579-. 

,936. 

','802'if 

793. 


— TTCC 

ACC  8.594 

6CC 

ACC  2.788  . , 

ACC 7,  65T— 7793  , 

UMACO  9,053  .391. 

-tJnACC 57x50 — 7315r 

ACC  5.931  .941._ 

-ACC 2,593— i875» 


8.984_488* 

— 67916 

17.982  ,0U» 

—777X3-79217- 

13,974  ,064. 

--p.981“X9Br 

5.^9  .207* 

8.l70  .809* 


iSe  Jo^. 

■ 6T48O—  ,125* 


re — 

XCC- 

9,772 — rwrr- 

-•9«6tl- 

■7«90» 

79 

unacc 

6.095  ,?7«. 

4,i85 

,/22* 

&0 

ACC- 

3,9^6  *5/5* 

“A^-iOV 

,425>" 

el 

ACC 

5.527  ,9A0» 

11.869 

,040* 

ez 

ACC- 

6,996  ,635# 

■"‘BTiOO 

“7365* 

83 

ACC 

2.529  ,542» 

2.464 

,458* 

a« 

ACC~ 

■ 6,977  ,717, 

■728JI 

65 

UMACC 

21,828  ,253* 

19.662 

,/47* 

- 6S 

ACC 

3,669  ,839* 

6.968 

■;i6iT-  ■ 

«7 

ACC 

7,328  ,897, 

11.661 

,103* 

6B 

ACC- 

■ 2,888-  ,9C6. 

‘ 7;430“ 

■,094*—  - 

89 

ACC 

13,101  ,882, 

17.115 

,118* 

90 

9,160  iU62t 

<3.a59 

91 

ACC 

10.238  ,948* 

16.o30 

,052* 

9Z 

ACC 

3,522  ^,953* 

9.660 

‘,04TA“  ■ 

93 

UNACC 

9.537  ,476* 

9.343 

,524* 

9«» 

ACC 

7976* 

11.569 

,024* 

95 

ACC 

1.625  ,742* 

3.792 

,258* 

96 

ACC" 

9.312  ‘,909* 

■“BTgZO 

,D9T* 

97 

ACC 

5.943  ,988* 

14.753 

,012» 

'9« 

SCC“ 

9,24B  ,91b* 

,090* 

99 

ACC 

10.305  ,993* 

20.336 

,007* 

lOO 

OnACC 

“^913  ,389, 

6.0IO 

,6n» 

(,ROUp  TCg  CJmSC^ 

UNACt 

— case ^ 


1 

unacc 

8.732 

•J05*_ 

4.448  .895* 

2 

UnACC  8.501 

. ,254* 

‘6.348  ”.746*» 

3 

. ACC 

3,365 

.670* 

7,i72  ,130» 

4 

DMACC~ 

■19,319" 

.248* 

IZ.gVfl  .752^ 

5 

J.MACC 

18,899 

.014* 

10.326  .986» 

6 

ACC 

6.136' 

.ooi*  ” 

“8.915  .199* 

7 

u.nacc 

8,189 

.407* 

7.432  ,593* 

8 

WACC  T.375  ^,l3l. 

■3.599  .869* ‘ 

9 

UMACC 

8.380 

.057* 

2.779  .943* 

10 

— onicc- 

6,146 

.255*“ 

3.406  ,7'#6* 

11 

14,334 

.018* 

6.p84  .982* 

12 

DmACC“ 

“11708^“ 

.176*" 

8.o09‘  ,824*  ' 

13 

ACC 

8.428 

.553* 

8,«57  ,947» 

19 

DvACC- 

— 4,0B3~ 

"iSlTc 

"r.s52  - ;683~~ 

15 

UMACC 

10.296 

.073* 

5,224  ,927* 

* 16 

ACC 

37559“ 

. 638  * 

STbSCT  ,762> 

17 

U'lACC 

3.169 

.483* 

3,n33  ,517* 

18 

URACC” 

■I3,o7Z~ 

.097* 

‘7,336' ',953* 

19 

ACC 

4,378 

.897* 

8,712  ,103* 

2or 

UfJACC“ 

B~Z76~' 

■.332'*" 

6.^74”  ,668* 

21 

unacc 

21.418 

.014* 

12,874  ,986* 

22 

UKiAcC 

4,0DJ 

.290* 

2.064  t7iir» 

23 

UNACC 

5,219 

.165* 

1.960  .835* 

■ ACC 

— X',720“ 

.787* 

4.335  ,213* 

25 

ACC 

6,050 

.504* 

6,r,83  ,'♦95* 

“26 

- ACC- 

~Z.46T' 

■.559*‘“ 

'2.937  ,44P 

27 

UNACC 

7,887 

.097* 

3,425  .903* 

26 

Una(«^ 

— 9;  815' 

TT47T- 

evyi  ,859* 

29 

UmACC 

8.545 

.069* 

3,3ti  ,931* 

3B 

UNACC 

'V'.SOS 

■;3i8V 

7i974  ,682* 
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32  UNACC  14.696  .032 

— 33 lKACC---  t),306  ~3ft6 

34  UNACC  13,466  .Q46 

35 OrJACC— 14;S42 — -,gl7 

36  U.-4ACC  8.121  .213 

37 OTKCC — 13-.017 — TfflT 

38  UNACC  10.284  .069 

29 0?IACC 6,-876 — TI27 

40  UNACC  8,553  .443 

— 41 ujjAcc  “10.927— res 

42  UNACC  11.202  ..044 

43 onAcc — — TTUr 

44  ACC  4.651  .870 

45 UriACC »-.-9Br--73 13" 

46  UNACC  12.170  .058 

■ “47 DTIACC — 107449 — ; 156" 

48  UNACC  13,391  .cl8 

49  U'lACC  8,851  H'TT 

50  UfJACC  12,816  .?21 

51 ACC 47771— T7T2“ 

52  UNACC  13.222  .cib 

Tj3 TJflACC  “19;5S7~7  6'19 

54  ACC  2.696  .666 

55 OlACC — 15.334 — 

56  UNACC  9,758  .158 

57 -7TCC — 0.*972~;t,2i 

58  UNACC  7,971  ,209 

59 OTiACC 6;495“~.4&2 

60  UNACC  5.688  ,i79 

“TBl W7ACC — 1*7888 — 7CST 

62  ACC  2.604  ,515 

— 63 D;^ACC"1*;5I7 — 

64  UUACC  15.050  ,C48 

—65 VrtCC 77449 — Tf57' 

66  ACC  5.925  .813 

— 67 crmcc — 1*7830 — ixfirr 

t8  UNACC  9,231  ,H14 

69 triACC 9, 029— ,-163 

70  UNACC  6,313  ,i56 

71 DHACC — rO;933  -.ISr 

72  UNACC  15,957  ,c30 

73 CTWCC — 1*7791 7C?r 

74  UnACC  6.430  .436 

— 75- D»TACC 4, 658 — ;?5i" 

76  UNACC  6,211  .328 


60 

■81 — 
82 

“cr- 

84 


UNACC  8,860 

ACC 57752' 

UNACC  11,849 

UyjACC 8-.290 

UNACC  10,902 
ONACC— 30. 836T 
UNACC  6,569 

ottacc — 97797- 

UNACC  4,348 
DHACC— 15 ,379 ' 


7.a78  ,968» 

- 4.a79  ■“.614# — 

7. „11  ,954» 

“6.471  ■, 983# — 

5.5O7  ,787# 

— 4T9S2 — ;983» — 
5.068  ,931» 

"■3;oI6"~.873» 

8.o91  ,557» 

-T.686— 7«35»~ 
5.066  .956# 

— 3VT25— .-«98» 

8.460  .130^ 

-77416—687# — 
6.h81  .942# 

~7.p99  -.842»  - 

5.357  .982# 

•“77571—7^7# 

IO.3OO  .779# 

“6778r~  .2687 

4.?90  .985# 

11.742  “;98p — 
4.r,79  ,334# 

"BTllC — 798117 

6.4O5  .842» 

- 4.141"  -,4797— 
5.5I2  ,791» 

"5;700'  .59A, 

3.793  .721# 

—87306— ,-908# 

2.722  .485# 

“ 6.-333  .957#-  - 
9.o77  .952# 

“47n89'  .843# 

e,c)69  ,187» 

—77096 — 7*38# 

8. ^38  .886# 

-5.751--  ,837r-  - 

2.q3l  ,844* 

-"r."s95  -.845* — 
9.0OI  ,970* 


5,914  ,564* 

~r,473— 7749*  • 
4.777  .672* 

-- 5ri3T— .982*  -- 
3.396  ,939* 

— 57ii42~7'»6in — 

6.rt62  ,923* 

- 2.554  “,946*- 

6.687  ,892# 

21.798  ■ .984*  — 
3.795  .800* 

—77558—7705# 

2,nie  ,682# 


\ 
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BB 

69 

u?wcc- 

UriACC 

•I0;S53~ 

10.870 

•090. 

•965. 

— 0T?79- 
10*s69 

-;9so» 

,535» 

90 

91 

OtWCC- 

UlACC 

3;6rr 

16.069 

*996. 

•092. 

9«h03 

,50¥. 

,958» 

92 

93 

*cc  ^.Tiir 
L'NACC  5.602 

,666. 

*122. 

9*0«6  *«9» 

1*6*»6  *678* 

UMACV 

5,917 

•325. 

9*%53 

,575* 

95 

ACC 

3.327 

,797. 

6*q65 

,203* 

96 

OmACC  51707— ,328. 

9,p70 

,672? 

97 

ACC 

6.098 

,679. 

10*o67 

,121* 

98 

DnSCC” 

”13^635“ 

*030. 

6*fc59 

^0* 

99 

ACC 

6.236 

*609. 

7*,89 

,396* 

100 

OlWCC' 

TO, 329" 

*099. 

9*  \lt 

GROOP 

*cc 

DN*CC 


CLAS^iPICb  l>?r6  GROyp  •. 

hCC  UR/>CC 
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«jTEP  variable  f value  TO  NUmBER  OF  U-STATISTIC 
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